From 7172278c9c9aeca28aee1e1c7740434336f7123f Mon Sep 17 00:00:00 2001 From: Rachael Graham Date: Mon, 12 Oct 2026 10:51:50 -0500 Subject: [PATCH 01/10] Document multi-actor Worker capacity Update Agent Substrate docs to describe Workers hosting multiple Actors, placement and capacity limits, pool scaling metrics, and snapshot storage options. Signed-off-by: Rachael Graham --- .../snippets/snapshot-cycle-diagram.md | 2 +- .../1.x/about/architecture/agent-substrate.md | 37 ++++++++++++++++--- .../1.x/about/substrate-runtime/sandboxing.md | 2 +- .../substrate-runtime/suspend-and-resume.md | 8 ++-- .../1.x/operations/tune-agent-substrate.md | 12 ++++-- docs-site/content/kagent/1.x/reference/faq.md | 2 +- 6 files changed, 47 insertions(+), 16 deletions(-) diff --git a/docs-site/assets/kagent-docs/snippets/snapshot-cycle-diagram.md b/docs-site/assets/kagent-docs/snippets/snapshot-cycle-diagram.md index b52db498..66a3b941 100644 --- a/docs-site/assets/kagent-docs/snippets/snapshot-cycle-diagram.md +++ b/docs-site/assets/kagent-docs/snippets/snapshot-cycle-diagram.md @@ -3,7 +3,7 @@ flowchart LR pool["WorkerPool"] --> worker1["Worker"] worker1 -->|hosts| actor["Actor
(running)"] actor -->|suspend| snapshot["ActorSnapshot
(immutable)"] - snapshot -->|resume| worker2["Any free Worker
in the pool"] + snapshot -->|resume| worker2["Any Worker with room
in the pool"] snapshot -->|pinned by| tag["ActorSnapshotTag
(retention pin)"] tag -->|seeds| newactor["New Actor"] ``` diff --git a/docs-site/content/kagent/1.x/about/architecture/agent-substrate.md b/docs-site/content/kagent/1.x/about/architecture/agent-substrate.md index 20c7ec62..2ed7f8f6 100644 --- a/docs-site/content/kagent/1.x/about/architecture/agent-substrate.md +++ b/docs-site/content/kagent/1.x/about/architecture/agent-substrate.md @@ -17,17 +17,42 @@ Substrate rejects any change to an ActorTemplate's spec after it is created. Thi ## Workers and WorkerPools -An Actor needs somewhere to run. Each Actor runs on a **Worker**, a pre-started pod that waits to receive one. Instead of starting a new pod each time a Session needs an Actor, Substrate schedules that Actor onto a Worker that is already running. The sandbox boundary sits around the Actor, not around the Worker pod that hosts it. +An Actor needs somewhere to run. Each Actor runs on a **Worker**, a pre-started pod that Substrate places Actors onto. Instead of starting a new pod each time a Session needs an Actor, Substrate schedules that Actor onto a Worker that is already running. The sandbox boundary sits around the Actor, not around the Worker pod that hosts it. - +### WorkerPools Workers come from a **WorkerPool**, a Kubernetes custom resource that an operator provisions before any Agent can compile. A WorkerPool declares how many Workers to keep running and which sandbox technology those Workers use. An operator never creates a Worker directly. Substrate manages them, keeping enough ready in each WorkerPool so that an Actor can start or resume on one immediately, without waiting on the Kubernetes scheduler to place a new Pod. +A WorkerPool exposes a Kubernetes scale subresource. A HorizontalPodAutoscaler can use it to size a pool from a metric instead of a fixed replica count. Agent Substrate publishes how many Workers in each pool have no free Actor slot. An autoscaler that watches that number adds Workers before the pool fills, rather than after placements start to fail. + + + + +### Worker capacity + +One Worker hosts many Actors at the same time. When each Worker starts, it reports a maximum capacity of Actors that it can host. A Worker that reaches that capacity refuses any further Actors, and Substrate puts those Actors on another Worker in the pool. The Worker's Actor capacity is a property of the Worker build, not a setting on the pool. The gVisor and micro-VM Workers that Agent Substrate ships both report a capacity of 1000 Actors. + +### Actor placement + +Substrate checks two things before it places an Actor on a Worker: +* Whether the Worker has a free Actor slot. This check always runs. +* Whether the Worker has enough spare CPU and memory. This check runs only when the ActorTemplate declares what one Actor needs, and kagent declares nothing for a Session. The free Actor slot check is therefore the only one that decides where a Session's Actor lands. [Worker resource limits]({{< link path="operations/tune-agent-substrate#set-resource-requests-on-workers" >}}) still matter, because a Worker with no limits reports its whole node as its own capacity. + + + +To determine where an Actor is placed, Substrate first narrows the Worker pool to the Workers that can take the Actor. It then picks two of those Workers at random and places the Actor on the less loaded of the two. Substrate determines a Worker's load by picking either the share of its Actor slots that are filled or the share of its CPU and memory that is committed, depending on which is higher. Sampling two Workers rather than ranking the whole pool keeps the decision cheap, and it avoids the pile-up that follows when every placement aims at whichever Worker looks emptiest at that moment. + + + ## Atespaces -An **atespace** is the isolation boundary that an Actor belongs to, and the first half of its identity. Agent Substrate addresses an Actor by its atespace and its name together, so the same Actor name can exist in two atespaces without colliding. Although an atespace resembles a Kubernetes namespace, it is not one. Agent Substrate defines it as its own cluster-wide resource, so no namespace contains it, a namespace's RBAC does not reach it, and deleting a Kubernetes namespace does not delete the atespace that kagent named after it. Actors and Workers sit differently against that boundary: an Actor belongs to exactly one atespace for its entire life, and a Worker belongs to none, because a Worker hosts whichever Actor Substrate places on it. +An **atespace** is the isolation boundary that an Actor belongs to, and the first half of its identity. Agent Substrate addresses an Actor by its atespace and its name together, so the same Actor name can exist in two atespaces without colliding. + +An atespace resembles a Kubernetes namespace, but it is not one. Agent Substrate defines it as its own cluster-wide resource. No namespace contains it, a namespace's RBAC does not reach it, and deleting a Kubernetes namespace does not delete the atespace that kagent named after it. + +Actors and Workers sit differently against this boundary. An Actor belongs to exactly one atespace for its entire life, whereas a Worker belongs to none. A Worker hosts whichever Actors Substrate places on it, and those Actors can come from different atespaces. kagent names each atespace after the Kubernetes namespace of the Agent whose Actor it holds, and creates that atespace on demand the first time a Session in the namespace needs an Actor. The Actor's own name is `session-` followed by the Session's identifier. A Session on an Agent in the `kagent` namespace therefore runs on an Actor that Agent Substrate addresses within the `kagent` atespace. Both halves of that identity appear in the address that traffic uses to reach the Actor, which [Sandboxing]({{< link path="about/substrate-runtime/sandboxing#how-traffic-reaches-a-sandboxed-actor" >}}) covers. @@ -39,18 +64,18 @@ kagent compiles each ActorTemplate to the sandbox class of the WorkerPool that h ## Suspend, snapshot, and resume -Substrate's density model rests on one fact about agent workloads: an Actor spends most of its time idle, waiting on a person or a large language model (LLM) to respond, not actively computing. Substrate exploits that by suspending idle Actors and reclaiming their Worker, then resuming them on demand when traffic arrives. Suspending and resuming allows a WorkerPool to run far more Actors than it has Workers for at any given moment. +Substrate's density model rests on one fact about agent workloads: an Actor spends most of its time idle, waiting on a person or a large language model (LLM) to respond, not actively computing. Substrate exploits that by suspending idle Actors and reclaiming the capacity that they held, then resuming them on demand when traffic arrives. Suspending and resuming allows a WorkerPool to run far more Actors than it has Workers for at any given moment. The following diagram traces an Actor through one suspend-and-resume cycle, and shows the second path that opens up once the resulting snapshot is tagged.

{{< reuse "kagent-docs/snippets/snapshot-cycle-diagram.md" >}} -A **WorkerPool** keeps **Workers** running and ready, and one Worker hosts the **Actor** while its conversation is active. Suspending that Actor writes its full state to an immutable **ActorSnapshot** and frees the Worker that it was running on. +A **WorkerPool** keeps **Workers** running and ready, and a Worker hosts the **Actor** alongside other Actors while its conversation is active. Suspending that Actor writes its full state to an immutable **ActorSnapshot** and releases the capacity that it held on its Worker. The diagram forks at that snapshot, because a snapshot serves two purposes. -- **Resume** restores the same Actor onto **any free Worker in the pool**, which is not necessarily the Worker that it ran on before. Because the snapshot captures the Actor's full state, the conversation continues from where it left off. Every idle agent takes this path. +- **Resume** restores the same Actor onto **any Worker in the pool with room**, which is not necessarily the Worker that it ran on before. Because the snapshot captures the Actor's full state, the conversation continues from where it left off. Every idle agent takes this path. - A **{{< gloss "Tag" >}}Tag{{< /gloss >}}** pins that snapshot, and a **New Actor** can be seeded from the tag at the moment that it is created. Resuming an existing Actor never goes through a tag. A tag gives a snapshot a stable, human-meaningful name, so callers do not need to track Substrate's internal snapshot identity. A tag names one snapshot permanently, and only its visibility scope can change afterward. A tag also acts as a retention pin, so Substrate does not delete a snapshot while a tag still names it. diff --git a/docs-site/content/kagent/1.x/about/substrate-runtime/sandboxing.md b/docs-site/content/kagent/1.x/about/substrate-runtime/sandboxing.md index 16951b0c..09194180 100644 --- a/docs-site/content/kagent/1.x/about/substrate-runtime/sandboxing.md +++ b/docs-site/content/kagent/1.x/about/substrate-runtime/sandboxing.md @@ -7,7 +7,7 @@ aliases: - /kagent/1.x/substrate-runtime/sandboxing/ --- -An agent is a program that decides at run time what to do next. It runs the commands that a model asks for, and it calls the tools it can access. [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}) runs each agent inside an **Actor**, its own unit of compute, and it does not run that Actor as an ordinary container process. Each Actor runs inside its own **sandbox**, on a {{< gloss "Worker" >}}Worker{{< /gloss >}} that hosts one Actor at a time. This page explains what selects a sandbox, what the sandbox separates, and how traffic reaches an Actor through it. The same boundary holds a [standalone sandbox]({{< link path="about/substrate-runtime/standalone-sandboxes" >}}), a scratch environment that runs commands and holds files without an agent conversation owning it. +An agent is a program that decides at run time what to do next. It runs the commands that a model asks for, and it calls the tools it can access. [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}) runs each agent inside an **Actor**, its own unit of compute, and it does not run that Actor as an ordinary container process. Each Actor runs inside its own **sandbox**, on a {{< gloss "Worker" >}}Worker{{< /gloss >}} that hosts many Actors at the same time, each in a sandbox of its own. This page explains what selects a sandbox, what the sandbox separates, and how traffic reaches an Actor through it. The same boundary holds a [standalone sandbox]({{< link path="about/substrate-runtime/standalone-sandboxes" >}}), a scratch environment that runs commands and holds files without an agent conversation owning it.

```mermaid diff --git a/docs-site/content/kagent/1.x/about/substrate-runtime/suspend-and-resume.md b/docs-site/content/kagent/1.x/about/substrate-runtime/suspend-and-resume.md index 79f66fda..91919c04 100644 --- a/docs-site/content/kagent/1.x/about/substrate-runtime/suspend-and-resume.md +++ b/docs-site/content/kagent/1.x/about/substrate-runtime/suspend-and-resume.md @@ -7,13 +7,13 @@ aliases: - /kagent/1.x/substrate-runtime/suspend-and-resume/ --- -An agent spends most of its life waiting. It waits on a person to reply, and it waits on a large language model (LLM) to answer. [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}) runs each agent inside an **Actor**, the unit that it suspends and resumes, and it treats that idle time as reclaimable: it suspends an idle Actor into a {{< gloss "Snapshot" >}}snapshot{{< /gloss >}}, frees the {{< gloss "Worker" >}}Worker{{< /gloss >}} that the Actor was running on, and restores the Actor when traffic arrives for it. This page explains what a snapshot captures, when kagent suspends an Actor, and what happens when a suspended Actor is addressed again. +An agent spends most of its life waiting. It waits on a person to reply, and it waits on a large language model (LLM) to answer. [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}) runs each agent inside an **Actor**, the unit that it suspends and resumes, and it treats that idle time as reclaimable: it suspends an idle Actor into a {{< gloss "Snapshot" >}}snapshot{{< /gloss >}}, releases the capacity that it held on its {{< gloss "Worker" >}}Worker{{< /gloss >}}, and restores the Actor when traffic arrives for it. This page explains what a snapshot captures, when kagent suspends an Actor, and what happens when a suspended Actor is addressed again. ## Actor lifecycle operations {{< gloss "Agent Substrate" >}}Agent Substrate{{< /gloss >}} provides three lifecycle operations, and each one moves an Actor between states that you can observe on the Actor record. -- **Suspend**: Writes the Actor's state to a durable snapshot in snapshot storage, then frees its Worker. A running Actor is snapshotted on its Worker. A paused Actor's node-local snapshot is uploaded instead. +- **Suspend**: Writes the Actor's state to a durable snapshot in snapshot storage, then releases the capacity that it held on its Worker. A running Actor is snapshotted on its Worker. A paused Actor's node-local snapshot is uploaded instead. - **Pause**: Takes a short-term snapshot whose files stay on the node. Pausing pins the Actor to that node, because the following resume is prioritized onto the node that holds the snapshot files. - **Resume**: Restores a suspended or paused Actor onto a Worker, from its latest snapshot. The common path restores from a snapshot rather than cold-booting the workload. @@ -25,7 +25,7 @@ The following diagram traces an Actor through those operations, and shows the fu An Actor reports its position in that cycle through its state, which is one of `RESUMING`, `RUNNING`, `SUSPENDING`, `SUSPENDED`, `PAUSING`, `PAUSED`, `CRASHED`, or `DELETING`. Only a suspended Actor can be deleted. > [!NOTE] -> Resume restores an Actor onto whichever Worker in the pool is free, which is not necessarily the Worker that the Actor ran on before. Suspend and resume let a {{< gloss "WorkerPool" >}}WorkerPool{{< /gloss >}} carry far more Actors than it has Workers at any one moment. +> Resume restores an Actor onto whichever Worker in the pool has room, which is not necessarily the Worker that the Actor ran on before. Suspend and resume let a {{< gloss "WorkerPool" >}}WorkerPool{{< /gloss >}} carry far more Actors than it has Workers at any one moment. ## What a snapshot captures @@ -50,7 +50,7 @@ Two kinds of snapshot serve different purposes, and both appear in a normal inst - **Golden snapshot**: Captured once, when an ActorTemplate is created, from a temporary golden boot of the workload. Every Actor of that template is first restored from this one shared snapshot, so a new Actor starts from an already-booted image rather than a cold start. An {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}} is not ready until its golden snapshot exists. Until then, the kagent controller reports `waiting for the ActorTemplate golden snapshot` while Agent Substrate captures it. - **Last snapshot**: The most recent per-Actor snapshot, written on every suspend and used to restore that specific Actor on its next resume. Because it carries the Actor's own accumulated state, the conversation continues from where it stopped. -Snapshots are persisted to object storage, either Google Cloud Storage or Amazon Simple Storage Service (S3), so that Actor state is durable and portable across the cluster. A {{< gloss "Harness" >}}Harness{{< /gloss >}} names the location for its Actors' snapshots in its `substrate.snapshotPolicy` section. +Snapshots are persisted to object storage so that Actor state is durable and portable across the cluster. Agent Substrate writes snapshots to Google Cloud Storage or to Amazon Simple Storage Service (S3). The S3 client also reaches S3-compatible stores that accept a custom endpoint and path-style addressing. For the settings that select and configure each client, see [Configure snapshot storage]({{< link path="operations/tune-agent-substrate#configure-snapshot-storage" >}}). A {{< gloss "Harness" >}}Harness{{< /gloss >}} names the location for its Actors' snapshots in its `substrate.snapshotPolicy` section. ```yaml spec: diff --git a/docs-site/content/kagent/1.x/operations/tune-agent-substrate.md b/docs-site/content/kagent/1.x/operations/tune-agent-substrate.md index ed2c9a40..b9a24626 100644 --- a/docs-site/content/kagent/1.x/operations/tune-agent-substrate.md +++ b/docs-site/content/kagent/1.x/operations/tune-agent-substrate.md @@ -80,7 +80,11 @@ A Worker entry gains an `actorId` only while an Actor occupies it, so the Worker The replica count is the only capacity dial on a pool, and setting it well depends on knowing what actually consumes a Worker. -A {{< gloss "Worker" >}}Worker{{< /gloss >}} hosts at most one Actor at a time, and it holds that Actor only while a turn is running. kagent suspends an Actor at every turn boundary and frees its Worker, as described in [Suspend and resume]({{< link path="about/substrate-runtime/suspend-and-resume" >}}). An idle {{< gloss "Session" >}}Session{{< /gloss >}} therefore occupies no Worker at all. +A {{< gloss "Worker" >}}Worker{{< /gloss >}} hosts many Actors at the same time, and it holds each one only while a turn is running. kagent suspends an Actor at every turn boundary and frees the Actor's share of its Worker, as described in [Suspend and resume]({{< link path="about/substrate-runtime/suspend-and-resume" >}}). An idle {{< gloss "Session" >}}Session{{< /gloss >}} therefore occupies no Worker capacity at all. + +Each Worker reports an Actor capacity when it starts, fixed by the Worker build rather than by a field on the WorkerPool. The gVisor and micro-VM Workers that Agent Substrate ships both report a capacity of 1000 Actors, and no setting raises or lowers it. + +Agent Substrate checks this capacity on every placement. It weighs a Worker's spare CPU and memory only when the ActorTemplate states what one Actor needs, and the templates that kagent generates for Sessions state nothing. A pool therefore places a Session's Actor based only on a Worker's available slots, regardless of how loaded the Worker's node already is. Worker resource limits still matter, because they govern how Kubernetes packs Workers onto nodes. Size the pool for the number of turns that run at the same time, not for the number of Agents or Sessions that you have created. A cluster with hundreds of Sessions that are each used occasionally needs far fewer Workers than the session count suggests. @@ -99,7 +103,9 @@ substrateWorkerPool: ### Recognize an undersized pool -A pool with too few Workers does not report a capacity error anywhere. Turns that cannot get a Worker wait for one, and a turn that waits too long fails with a generic timeout that names neither capacity nor the pool. +Agent Substrate reports pool capacity through its metrics rather than through an error that reaches the caller. Each Worker publishes a state of `idle`, `partial`, `at_capacity`, or `unschedulable`, which together show how much of a pool is in use, and a placement that finds no suitable Worker records a `no_capacity` outcome. Agent Substrate treats that outcome as a normal result rather than a failure, so it carries no error type and raises no alert on its own. Watch the share of Workers at `at_capacity` and the rate of `no_capacity` placements to tell a pool that is merely busy from one that is too small. + +None of those signals reach the caller. A turn that cannot get a Worker waits for one, and a turn that waits too long fails with a generic timeout that names neither capacity nor the pool. ```console ERROR: @@ -107,7 +113,7 @@ ERROR: Message: actor "ai-01a087c0-1d72-775a-9a32-566acac7b685" request timed out ``` -Because short turns release their Worker quickly, a pool under mild pressure absorbs the load and only sheds the requests that wait past the deadline. The result is intermittent timeouts under concurrency rather than a clean failure. Neither the kagent controller log nor the Agent Substrate API log records a capacity message when this happens, so diagnose it from the pool's `READY` count and raise the replica count. +Because short turns release their capacity quickly, a pool under mild pressure absorbs the load and only sheds the requests that wait past the deadline. The result is intermittent timeouts under concurrency rather than a clean failure. Neither the kagent controller log nor the Agent Substrate API log records a capacity message when this happens, so confirm the diagnosis from the `at_capacity` and `no_capacity` signals and from the pool's `READY` count, then raise the replica count. ### Set resource requests on Workers diff --git a/docs-site/content/kagent/1.x/reference/faq.md b/docs-site/content/kagent/1.x/reference/faq.md index f059ced8..bed49bd1 100644 --- a/docs-site/content/kagent/1.x/reference/faq.md +++ b/docs-site/content/kagent/1.x/reference/faq.md @@ -28,7 +28,7 @@ A {{< gloss "Session" >}}Session{{< /gloss >}} is one running conversation with ## How does suspend and resume work? -Agent conversations are mostly idle, so Agent Substrate does not hold a pod open between turns. When a turn ends, it writes the Actor's memory and filesystem to a {{< gloss "Snapshot" >}}snapshot{{< /gloss >}} and releases the {{< gloss "Worker" >}}Worker{{< /gloss >}} that was hosting it. The next message restores that snapshot onto whichever Worker is free, and the conversation continues where it stopped. [Suspend and resume]({{< link path="about/substrate-runtime/suspend-and-resume" >}}) covers the lifecycle, what a snapshot captures, and how {{< gloss "Checkpoint" >}}checkpoints{{< /gloss >}} pin one. +Agent conversations are mostly idle, so Agent Substrate does not hold a pod open between turns. When a turn ends, it writes the Actor's memory and filesystem to a {{< gloss "Snapshot" >}}snapshot{{< /gloss >}} and releases the capacity that it held on its {{< gloss "Worker" >}}Worker{{< /gloss >}}. The next message restores that snapshot onto whichever Worker has room, and the conversation continues where it stopped. [Suspend and resume]({{< link path="about/substrate-runtime/suspend-and-resume" >}}) covers the lifecycle, what a snapshot captures, and how {{< gloss "Checkpoint" >}}checkpoints{{< /gloss >}} pin one. ## How is 1.0 different from 0.x? From c0b28f31472043740ed63792410dcd7497e12a85 Mon Sep 17 00:00:00 2001 From: Rachael Graham Date: Mon, 12 Oct 2026 11:14:58 -0500 Subject: [PATCH 02/10] Clarify sandboxing and network flow docs Explain the agentic loop, how sandbox classes and SandboxConfigs are selected, and how requests and network policies flow through the runtime. Add a review note flagging uncertainty about custom egress destinations. Signed-off-by: Rachael Graham --- .../1.x/about/substrate-runtime/sandboxing.md | 24 ++++++++++++------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/docs-site/content/kagent/1.x/about/substrate-runtime/sandboxing.md b/docs-site/content/kagent/1.x/about/substrate-runtime/sandboxing.md index 09194180..1ae4ea29 100644 --- a/docs-site/content/kagent/1.x/about/substrate-runtime/sandboxing.md +++ b/docs-site/content/kagent/1.x/about/substrate-runtime/sandboxing.md @@ -7,7 +7,11 @@ aliases: - /kagent/1.x/substrate-runtime/sandboxing/ --- -An agent is a program that decides at run time what to do next. It runs the commands that a model asks for, and it calls the tools it can access. [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}) runs each agent inside an **Actor**, its own unit of compute, and it does not run that Actor as an ordinary container process. Each Actor runs inside its own **sandbox**, on a {{< gloss "Worker" >}}Worker{{< /gloss >}} that hosts many Actors at the same time, each in a sandbox of its own. This page explains what selects a sandbox, what the sandbox separates, and how traffic reaches an Actor through it. The same boundary holds a [standalone sandbox]({{< link path="about/substrate-runtime/standalone-sandboxes" >}}), a scratch environment that runs commands and holds files without an agent conversation owning it. +An agent runs an **agentic loop**. A model proposes the next step, the agent carries it out by running a command or calling a tool, and the result feeds the model's next decision. The loop repeats until the task finishes or the agent stops to wait on input from a person. + +kagent runs an agent's loop inside an **Actor**, a unit of compute that [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}) creates and manages. An Actor is not an ordinary container process. Each Actor runs inside its own **sandbox**, and one {{< gloss "Worker" >}}Worker{{< /gloss >}} hosts many of those sandboxed Actors at the same time. + +This page explains what selects a sandbox, what the sandbox separates, and how traffic reaches an Actor through it.

```mermaid @@ -32,18 +36,18 @@ A **sandbox class** is the sandbox runtime family that a Worker uses. {{< gloss - **`gvisor`** (default): Runs the workload against a [gVisor](https://gvisor.dev) user-space kernel, which keeps the workload's system calls from reaching the host kernel. - **`microvm`**: Runs the workload inside a lightweight virtual machine, which places a hypervisor boundary between the workload and the host. -A {{< gloss "WorkerPool" >}}WorkerPool{{< /gloss >}} selects its class through the `sandboxClass` field, which defaults to `gvisor`. The choice is not only a runtime preference. It also shapes the Worker pods that Agent Substrate creates for that pool, including the virtualization device mounts and node placement that a micro-VM needs. +A {{< gloss "WorkerPool" >}}WorkerPool{{< /gloss >}}'s `sandboxClass` field sets the sandbox class for its Workers, and defaults to `gvisor`. That value does more than name a runtime. Agent Substrate builds the pool's Worker pods to match it, mounting the virtualization devices that a micro-VM needs and placing those pods on nodes that provide them. > [!NOTE] -> kagent generates {{< gloss "ActorTemplate" >}}ActorTemplates{{< /gloss >}} that use the sandbox class of the pool that hosts them, and names the matching SandboxConfig on each template: `gvisor-default` for the `gvisor` class, `microvm` for the `microvm` class. A WorkerPool that backs kagent Harnesses must run the class it is meant to serve, with a matching Worker build. +> kagent copies a pool's sandbox class onto every {{< gloss "ActorTemplate" >}}ActorTemplate{{< /gloss >}} that it generates for that pool. Each template then names the SandboxConfig that serves its class. A `gvisor` template names `gvisor-default`, and a `microvm` template names `microvm`. ## Sandbox configuration -A **SandboxConfig** is a cluster-scoped resource that holds the material needed to start one sandbox runtime family. It carries the runtime assets that the node agent fetches, keyed by processor architecture, along with the pause image that holds the sandbox's namespaces as its root container. Each ActorTemplate names the configuration that it uses, and the name is required. Agent Substrate resolves no cluster default, so the configuration that a template names must exist before that template can be prepared. +A **SandboxConfig** is a cluster-scoped resource that describes the runtime files a sandbox class requires to start. The resource stores none of those files. Instead, it records the location that the node agent fetches each file from, keyed by processor architecture. It also names the pause image, which holds the sandbox's namespaces as its root container. -Defining these assets in a cluster resource lets one configuration pin a runtime version for many ActorTemplates at once, rather than each template carrying its own copy. +Every ActorTemplate must name a SandboxConfig, and Agent Substrate supplies no name of its own. A template cannot be prepared until the configuration that it names exists in the cluster. Many ActorTemplates can share one SandboxConfig. The runtime version is then pinned in a single place, rather than repeated on every template. -A default installation creates a single `gvisor-default` configuration, and kagent names exactly that configuration on every ActorTemplate that it generates. The configuration looks like the following: +A default installation creates one SandboxConfig, named `gvisor-default`. kagent writes that name onto every ActorTemplate that it generates, so each template carries it explicitly. The configuration looks like the following: ```yaml apiVersion: ate.dev/v1alpha1 @@ -89,12 +93,16 @@ Every Actor is addressed by its {{< gloss "Atespace" >}}atespace{{< /gloss >}} a Because the router resolves the Worker assignment on every request, an Actor keeps a stable address across suspends, resumes, and moves between Workers. +A kagent user never uses that address. A message sent to a {{< gloss "Session" >}}Session{{< /gloss >}} through the kagent API or the web console reaches the kagent controller, which resolves the Session to its Actor and passes the request to Agent Substrate. Every hop after that stays inside the runtime. + Traffic in the other direction leaves through a separate egress gateway rather than going straight out from the Worker. Routing Actor egress through one gateway provides a single place to apply outbound controls. ## Default network posture Agent Substrate creates a Kubernetes NetworkPolicy for each WorkerPool, selecting that pool's Worker pods. The policy restricts **ingress** to the Agent Substrate router alone. No other pod in the cluster can open a connection to a Worker, so an Actor is not reachable by anything that bypasses the routing path. -That policy governs inbound traffic only. Outbound traffic is governed separately, by an egress policy that the gateway enforces on every connection that an Actor opens. That policy is default-deny: an Actor reaches a destination only when a rule allows it, and an Actor with no policy at all gets no outbound connection. +That policy governs inbound traffic only. Outbound traffic is governed separately by an egress policy that the gateway enforces on every connection that an Actor opens. That policy is default-deny. An Actor reaches a destination only when a rule allows it, and the gateway denies the connection when no rule does. + +kagent derives the egress policy from the {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}}, so an agent carries no egress configuration of its own. The destinations it allows come from what the AgentTemplate already names, such as its model provider and its MCP servers. You author neither policy. Agent Substrate owns the ingress policy, and kagent regenerates the egress policy whenever the AgentTemplate changes. For what the policy holds, how a rule is matched, and how to diagnose a denied request, see [Egress control]({{< link path="about/substrate-runtime/networking-and-egress" >}}). -kagent writes that policy for you, deriving it from the AgentTemplate, so an agent needs no egress configuration of its own. For what the policy holds, how a rule is matched, and how to diagnose a denied request, see [Egress control]({{< link path="about/substrate-runtime/networking-and-egress" >}}). + From 35d35fc0bc05cd50734a7facef8f028706a4d523 Mon Sep 17 00:00:00 2001 From: Rachael Graham Date: Mon, 12 Oct 2026 12:21:45 -0500 Subject: [PATCH 03/10] Update suspend-and-resume.md Signed-off-by: Rachael Graham --- .../substrate-runtime/suspend-and-resume.md | 33 ++++++++++--------- 1 file changed, 18 insertions(+), 15 deletions(-) diff --git a/docs-site/content/kagent/1.x/about/substrate-runtime/suspend-and-resume.md b/docs-site/content/kagent/1.x/about/substrate-runtime/suspend-and-resume.md index 91919c04..22757487 100644 --- a/docs-site/content/kagent/1.x/about/substrate-runtime/suspend-and-resume.md +++ b/docs-site/content/kagent/1.x/about/substrate-runtime/suspend-and-resume.md @@ -7,7 +7,7 @@ aliases: - /kagent/1.x/substrate-runtime/suspend-and-resume/ --- -An agent spends most of its life waiting. It waits on a person to reply, and it waits on a large language model (LLM) to answer. [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}) runs each agent inside an **Actor**, the unit that it suspends and resumes, and it treats that idle time as reclaimable: it suspends an idle Actor into a {{< gloss "Snapshot" >}}snapshot{{< /gloss >}}, releases the capacity that it held on its {{< gloss "Worker" >}}Worker{{< /gloss >}}, and restores the Actor when traffic arrives for it. This page explains what a snapshot captures, when kagent suspends an Actor, and what happens when a suspended Actor is addressed again. +An agent spends most of its life waiting. It waits on a person to reply, and it waits on a large language model (LLM) to answer. [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}) runs each agent inside an **Actor**, the unit that it suspends and resumes. Rather than hold compute open through that waiting, Agent Substrate writes the idle Actor to a {{< gloss "Snapshot" >}}snapshot{{< /gloss >}} and releases the capacity that the Actor held on its {{< gloss "Worker" >}}Worker{{< /gloss >}}. The next message restores it. The {{< gloss "Session" >}}Session{{< /gloss >}} stays ready the whole time, so reclaiming that capacity costs the user nothing. This page explains what a snapshot captures, when kagent suspends an Actor, and what happens when a suspended Actor is addressed again. ## Actor lifecycle operations @@ -25,7 +25,7 @@ The following diagram traces an Actor through those operations, and shows the fu An Actor reports its position in that cycle through its state, which is one of `RESUMING`, `RUNNING`, `SUSPENDING`, `SUSPENDED`, `PAUSING`, `PAUSED`, `CRASHED`, or `DELETING`. Only a suspended Actor can be deleted. > [!NOTE] -> Resume restores an Actor onto whichever Worker in the pool has room, which is not necessarily the Worker that the Actor ran on before. Suspend and resume let a {{< gloss "WorkerPool" >}}WorkerPool{{< /gloss >}} carry far more Actors than it has Workers at any one moment. +> Resume restores an Actor onto whichever Worker in the pool has room, which is not necessarily the Worker that the Actor ran on before. Because a suspended Actor occupies no Worker, a {{< gloss "WorkerPool" >}}WorkerPool{{< /gloss >}} can serve far more Sessions than it could if every Actor stayed resident. ## What a snapshot captures @@ -34,21 +34,22 @@ An {{< gloss "ActorTemplate" >}}ActorTemplate{{< /gloss >}}'s snapshot configura - **`Full`**: Captures process memory, the root filesystem changes layered on top of the container image, and any attached durable volumes. A `Full` snapshot holds everything that is needed to resume the Actor hot, with its in-memory state intact. - **`Data`**: Captures only the contents of attached durable volumes. Process memory and the rest of the root filesystem are discarded, which makes the snapshot much cheaper to write and store. -Scopes describe only what a snapshot captures, and they are configured per trigger. The `onPause` setting selects what a pause captures on the node, and `onCommit` selects what a suspend uploads to snapshot storage. What `onCommit` captures must be a subset of what `onPause` captures. +kagent sets these scopes automatically on every ActorTemplate that it generates. A pause captures `Full`, a commit captures `Data`, a resume cold-boots, and each Actor gets a single durable volume named `data`. The one snapshot setting that you author is the storage location, which you set on the {{< gloss "Harness" >}}Harness{{< /gloss >}}. + +A scope is set per trigger. The `onPause` setting selects what a pause captures on the node, and `onCommit` selects what a suspend uploads to snapshot storage. What `onCommit` captures must be a subset of what `onPause` captures. + +### Durable volumes and resume A **DurableDir volume** is the per-Actor application data surface. Its contents are preserved by the `Data` scope, so they survive a suspend and resume cycle independently of process memory. How many volumes an ActorTemplate can declare depends on its sandbox class. A `microvm` template can declare several, because they are subdirectories of a single shared filesystem. A `gvisor` template is limited to one, until {{< gloss "gVisor" >}}gVisor{{< /gloss >}} accepts more than a single durable mount. When an Actor resumes from a `Data`-scope snapshot, the ActorTemplate's `onResume.fromData` setting decides where the rest of the guest state comes from. The default is `ColdBoot`, which starts the containers fresh from the container image with the durable volume contents restored over them. -> [!NOTE] -> These scopes describe what Agent Substrate supports, not choices that you make. kagent compiles every ActorTemplate with the same snapshot configuration: `Full` on pause, `Data` on commit, `ColdBoot` on resume, and a single DurableDir volume named `data`. The only snapshot setting that you author is the storage location, on the Harness. - ## Golden and per-Actor snapshots Two kinds of snapshot serve different purposes, and both appear in a normal installation. -- **Golden snapshot**: Captured once, when an ActorTemplate is created, from a temporary golden boot of the workload. Every Actor of that template is first restored from this one shared snapshot, so a new Actor starts from an already-booted image rather than a cold start. An {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}} is not ready until its golden snapshot exists. Until then, the kagent controller reports `waiting for the ActorTemplate golden snapshot` while Agent Substrate captures it. -- **Last snapshot**: The most recent per-Actor snapshot, written on every suspend and used to restore that specific Actor on its next resume. Because it carries the Actor's own accumulated state, the conversation continues from where it stopped. +- The **golden snapshot** is shared by every Actor of one ActorTemplate. Agent Substrate captures it once by booting the workload temporarily when the template is created. Every new Actor of that template starts from this snapshot rather than from a cold start. kagent waits for the golden snapshot before an {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}} becomes ready, and the controller reports `waiting for the ActorTemplate golden snapshot` while the capture runs. +- The **last snapshot** belongs to one Actor. Agent Substrate writes it on every suspend and restores that same Actor from it on the next resume. It holds the files and memory that this one conversation built up, which is how the conversation continues from where it stopped. Snapshots are persisted to object storage so that Actor state is durable and portable across the cluster. Agent Substrate writes snapshots to Google Cloud Storage or to Amazon Simple Storage Service (S3). The S3 client also reaches S3-compatible stores that accept a custom endpoint and path-style addressing. For the settings that select and configure each client, see [Configure snapshot storage]({{< link path="operations/tune-agent-substrate#configure-snapshot-storage" >}}). A {{< gloss "Harness" >}}Harness{{< /gloss >}} names the location for its Actors' snapshots in its `substrate.snapshotPolicy` section. @@ -76,16 +77,18 @@ The {{< gloss "Session" >}}Session{{< /gloss >}}'s own state does not change whi ## Resuming on demand -Every Actor is reachable at a uniform address built from its {{< gloss "Atespace" >}}atespace{{< /gloss >}} and name, `..actors.resources.substrate.ate.dev`, resolved by Agent Substrate's own Domain Name System (DNS) server. Traffic sent to that name is routed to the right Worker, and an Actor that is currently suspended is resumed automatically to receive it. Nothing in the calling path needs to know whether the Actor was running beforehand. +A suspended Session resumes when the next message arrives for it. The kagent controller hands that message to Agent Substrate, which restores the Actor from its last snapshot onto a Worker with room, and then delivers the message. No caller needs to know whether the Actor was running beforehand, and nothing in kagent asks for the resume explicitly. -Resume speed makes suspending at every turn boundary practical rather than costly. Agent Substrate's own target for this cycle is 100 milliseconds at the ninety-fifth percentile, measured from the moment traffic arrives for a suspended Actor to the moment that Actor can receive it. +Agent Substrate resolves the Actor's address on every request, so the Actor stays reachable at one name no matter which Worker it is placed on. For more information, see [Sandboxing]({{< link path="about/substrate-runtime/sandboxing#how-traffic-reaches-a-sandboxed-actor" >}}). -## Checkpoints +Because kagent suspends at every turn boundary, every message to a Session pays for one resume. An idle Session costs no compute, and the next message waits for its Actor to come back. + + -A snapshot that Agent Substrate writes on suspend is transient. Agent Substrate is free to collect it once a newer snapshot supersedes it. A **checkpoint** makes one of those snapshots durable by pinning it. +## Checkpoints -Creating a checkpoint attaches an Agent Substrate {{< gloss "Tag" >}}Tag{{< /gloss >}} to the snapshot that the Session most recently suspended to. The tag names that one snapshot permanently and acts as a retention pin, such that Agent Substrate does not collect a snapshot while a tag still names it. Deleting the checkpoint removes the tag and releases the pin. +A **checkpoint** marks one point in a conversation and keeps it. You can create a checkpoint when a Session reaches a state worth returning to, such as a long investigation that is only partway through. You can then start a second Session from that point, which is known as a {{< gloss "Fork" >}}fork{{< /gloss >}}. The fork continues the conversation from where the checkpoint marked it, new turns append only to the fork, and the fork gets its own conversation and task identifiers rather than inheriting the source's. To create a checkpoint and fork a Session from it, follow the [Agent Substrate example]({{< link path="examples/agent-substrate" >}}). -A Session must be at a turn boundary to be checkpointed, because the turn boundary is captured. The request names the terminal task to pin. It fails when a newer turn has already started, and again while that turn's snapshot is still being written. +Checkpoint a Session only at a turn boundary, because the turn boundary is what gets captured. The request names the terminal task to pin, and it fails when a newer turn has already started or while that turn's snapshot is still being written. -A checkpoint also records how far the conversation had advanced, and it lets you start a second Session from the state it pinned. That second Session, a {{< gloss "Fork" >}}fork{{< /gloss >}}, continues the conversation from the point that the checkpoint pinned, and new turns append only to the fork. A fork receives fresh conversation and task identifiers rather than inheriting the source's. To create a checkpoint and fork a Session from it, work through the [Agent Substrate example]({{< link path="examples/agent-substrate" >}}). +Underneath, a checkpoint pins a snapshot that would otherwise be discarded. Agent Substrate treats the snapshot written on each suspend as transient and collects it once a newer one supersedes it. Creating a checkpoint attaches a {{< gloss "Tag" >}}Tag{{< /gloss >}} to the snapshot that the Session most recently suspended to, and Agent Substrate keeps any snapshot that a tag still names. Deleting the checkpoint removes the tag and releases the snapshot. From 501f8c346446f10758aedeed2d94fe95ee2a0edd Mon Sep 17 00:00:00 2001 From: Rachael Graham Date: Mon, 12 Oct 2026 13:11:24 -0500 Subject: [PATCH 04/10] egress Signed-off-by: Rachael Graham --- .../1.x/about/architecture/agent-substrate.md | 18 ++-- .../networking-and-egress.md | 92 ++++++++++++------- .../1.x/about/substrate-runtime/sandboxing.md | 14 +-- .../substrate-runtime/suspend-and-resume.md | 16 ++-- .../1.x/operations/tune-agent-substrate.md | 4 +- 5 files changed, 82 insertions(+), 62 deletions(-) diff --git a/docs-site/content/kagent/1.x/about/architecture/agent-substrate.md b/docs-site/content/kagent/1.x/about/architecture/agent-substrate.md index 2ed7f8f6..63b86451 100644 --- a/docs-site/content/kagent/1.x/about/architecture/agent-substrate.md +++ b/docs-site/content/kagent/1.x/about/architecture/agent-substrate.md @@ -11,13 +11,13 @@ aliases: ## ActorTemplate -An **ActorTemplate** is the definition that Substrate creates an Actor from. It carries what Substrate needs in order to start that Actor: the agent's container image, the command that starts the agent, the agent's environment, the sandbox class to isolate it with, and the WorkerPool to schedule it onto. The kagent controller produces an ActorTemplate by compiling an {{< gloss "Agent" >}}Agent{{< /gloss >}} and the resources that it names, so an ActorTemplate is where kagent's configuration model becomes executable by Substrate. +An **ActorTemplate** is the definition that Substrate creates an Actor from. It carries what Substrate needs in order to start this Actor: the agent's container image, the command that starts the agent, the agent's environment, the sandbox class to isolate it with, and the WorkerPool to schedule it onto. The kagent controller produces an ActorTemplate by compiling an {{< gloss "Agent" >}}Agent{{< /gloss >}} and the resources that it names, so an ActorTemplate is where kagent's configuration model becomes executable by Substrate. Substrate rejects any change to an ActorTemplate's spec after it is created. This immutability ensures that a running conversation remains on a fixed definition. The controller creates a new ActorTemplate for every compiled {{< gloss "Revision" >}}revision{{< /gloss >}} instead of editing an existing one, and it reclaims an old ActorTemplate once nothing references its revision. An Agent holds a reference for as long as the revision is its desired or its latest successful one, and a Session holds one for as long as it runs on that revision. ## Workers and WorkerPools -An Actor needs somewhere to run. Each Actor runs on a **Worker**, a pre-started pod that Substrate places Actors onto. Instead of starting a new pod each time a Session needs an Actor, Substrate schedules that Actor onto a Worker that is already running. The sandbox boundary sits around the Actor, not around the Worker pod that hosts it. +An Actor needs somewhere to run. Each Actor runs on a **Worker**, a pre-started pod that Substrate places Actors onto. Instead of starting a new pod each time a Session needs an Actor, Substrate schedules this Actor onto a Worker that is already running. The sandbox boundary sits around the Actor, not around the Worker pod that hosts it. ### WorkerPools @@ -25,14 +25,14 @@ Workers come from a **WorkerPool**, a Kubernetes custom resource that an operato An operator never creates a Worker directly. Substrate manages them, keeping enough ready in each WorkerPool so that an Actor can start or resume on one immediately, without waiting on the Kubernetes scheduler to place a new Pod. -A WorkerPool exposes a Kubernetes scale subresource. A HorizontalPodAutoscaler can use it to size a pool from a metric instead of a fixed replica count. Agent Substrate publishes how many Workers in each pool have no free Actor slot. An autoscaler that watches that number adds Workers before the pool fills, rather than after placements start to fail. +A WorkerPool exposes a Kubernetes scale subresource. A HorizontalPodAutoscaler can use it to size a pool from a metric instead of a fixed replica count. Agent Substrate publishes how many Workers in each pool have no free Actor slot. An autoscaler that watches this number adds Workers before the pool fills, rather than after placements start to fail. ### Worker capacity -One Worker hosts many Actors at the same time. When each Worker starts, it reports a maximum capacity of Actors that it can host. A Worker that reaches that capacity refuses any further Actors, and Substrate puts those Actors on another Worker in the pool. The Worker's Actor capacity is a property of the Worker build, not a setting on the pool. The gVisor and micro-VM Workers that Agent Substrate ships both report a capacity of 1000 Actors. +One Worker hosts many Actors at the same time. When each Worker starts, it reports a maximum capacity of Actors that it can host. A Worker that reaches this capacity refuses any further Actors, and Substrate puts those Actors on another Worker in the pool. The Worker's Actor capacity is a property of the Worker build, not a setting on the pool. The gVisor and micro-VM Workers that Agent Substrate ships both report a capacity of 1000 Actors. ### Actor placement @@ -54,7 +54,7 @@ An atespace resembles a Kubernetes namespace, but it is not one. Agent Substrate Actors and Workers sit differently against this boundary. An Actor belongs to exactly one atespace for its entire life, whereas a Worker belongs to none. A Worker hosts whichever Actors Substrate places on it, and those Actors can come from different atespaces. -kagent names each atespace after the Kubernetes namespace of the Agent whose Actor it holds, and creates that atespace on demand the first time a Session in the namespace needs an Actor. The Actor's own name is `session-` followed by the Session's identifier. A Session on an Agent in the `kagent` namespace therefore runs on an Actor that Agent Substrate addresses within the `kagent` atespace. Both halves of that identity appear in the address that traffic uses to reach the Actor, which [Sandboxing]({{< link path="about/substrate-runtime/sandboxing#how-traffic-reaches-a-sandboxed-actor" >}}) covers. +kagent names each atespace after the Kubernetes namespace of the Agent whose Actor it holds, and creates that atespace on demand the first time a Session in the namespace needs an Actor. The Actor's own name is `session-` followed by the Session's identifier. A Session on an Agent in the `kagent` namespace therefore runs on an Actor that Agent Substrate addresses within the `kagent` atespace. Both halves of this identity appear in the address that traffic uses to reach the Actor, which [Sandboxing]({{< link path="about/substrate-runtime/sandboxing#how-traffic-reaches-a-sandboxed-actor" >}}) covers. ## Sandboxing @@ -71,16 +71,16 @@ The following diagram traces an Actor through one suspend-and-resume cycle, and {{< reuse "kagent-docs/snippets/snapshot-cycle-diagram.md" >}} -A **WorkerPool** keeps **Workers** running and ready, and a Worker hosts the **Actor** alongside other Actors while its conversation is active. Suspending that Actor writes its full state to an immutable **ActorSnapshot** and releases the capacity that it held on its Worker. +A **WorkerPool** keeps **Workers** running and ready, and a Worker hosts the **Actor** alongside other Actors while its conversation is active. Suspending this Actor writes its full state to an immutable **ActorSnapshot** and releases the capacity that it held on its Worker. -The diagram forks at that snapshot, because a snapshot serves two purposes. +The diagram forks at this snapshot, because a snapshot serves two purposes. - **Resume** restores the same Actor onto **any Worker in the pool with room**, which is not necessarily the Worker that it ran on before. Because the snapshot captures the Actor's full state, the conversation continues from where it left off. Every idle agent takes this path. -- A **{{< gloss "Tag" >}}Tag{{< /gloss >}}** pins that snapshot, and a **New Actor** can be seeded from the tag at the moment that it is created. Resuming an existing Actor never goes through a tag. +- A **{{< gloss "Tag" >}}Tag{{< /gloss >}}** pins this snapshot, and a **New Actor** can be seeded from the tag at the moment that it is created. Resuming an existing Actor never goes through a tag. A tag gives a snapshot a stable, human-meaningful name, so callers do not need to track Substrate's internal snapshot identity. A tag names one snapshot permanently, and only its visibility scope can change afterward. A tag also acts as a retention pin, so Substrate does not delete a snapshot while a tag still names it. -For example, an agent partway through a long incident investigation reaches a state worth keeping. Creating a [checkpoint]({{< link path="about/substrate-runtime/suspend-and-resume#checkpoints" >}}) tags the snapshot that the agent most recently suspended to, which holds that one snapshot in place while the agent carries on and writes newer ones. Without the tag, Substrate collects that snapshot once a newer one supersedes it. +For example, an agent partway through a long incident investigation reaches a state worth keeping. Creating a [checkpoint]({{< link path="about/substrate-runtime/suspend-and-resume#checkpoints" >}}) tags the snapshot that the agent most recently suspended to, which holds that one snapshot in place while the agent carries on and writes newer ones. Without the tag, Substrate collects this snapshot once a newer one supersedes it. diff --git a/docs-site/content/kagent/1.x/about/substrate-runtime/networking-and-egress.md b/docs-site/content/kagent/1.x/about/substrate-runtime/networking-and-egress.md index 77188ac9..9308bf2b 100644 --- a/docs-site/content/kagent/1.x/about/substrate-runtime/networking-and-egress.md +++ b/docs-site/content/kagent/1.x/about/substrate-runtime/networking-and-egress.md @@ -7,79 +7,90 @@ aliases: - /kagent/1.x/substrate-runtime/networking-and-egress/ --- -An agent reaches the network only through Agent Substrate's egress gateway, and the gateway authorizes every connection against a policy attached to that {{< gloss "Actor" >}}Actor{{< /gloss >}}. The policy is default-deny: an Actor with no policy makes no outbound request at all. kagent builds the policy for you from the AgentTemplate. A documented agent therefore works without any egress configuration of its own. +An agent reaches the network only through Agent Substrate's egress gateway, and the gateway authorizes every connection against a policy attached to that {{< gloss "Actor" >}}Actor{{< /gloss >}}. The policy is default-deny. An Actor can attempt any request, and the gateway denies the ones that no rule allows. kagent derives the policy from the AgentTemplate, so a documented agent works without any egress configuration of its own. -> [!IMPORTANT] -> The gateway terminates TLS, so hostname rules constrain encrypted traffic as well as cleartext. An HTTPS request is decrypted at the gateway, matched against the policy's hostnames, and re-originated to the destination. An opaque TCP tunnel is the exception, because it carries no request for the gateway to read. For details, see [Traffic types](#traffic-types). +> [!NOTE] +> The gateway terminates TLS, so hostname rules constrain encrypted traffic as well as cleartext. An HTTPS request is decrypted at the gateway, matched against the policy's hostnames, and re-originated to the destination. A TLS passthrough rule is the exception, because the gateway forwards this connection without decrypting it. For details, see [Traffic types](#traffic-types). For the inbound half of the picture, and for the NetworkPolicy that protects Workers, see [Sandboxing]({{< link path="about/substrate-runtime/sandboxing#default-network-posture" >}}). ## How the gateway authorizes a connection -Outbound traffic leaves a Worker through the egress gateway rather than directly out. That single chokepoint gives Agent Substrate one place to identify the caller and apply rules, and the gateway does both on every connection. +Outbound traffic leaves a Worker through the egress gateway rather than directly out. This chokepoint gives Agent Substrate one place to decide what each agent can reach. The gateway takes three steps on every connection, and each step depends on the one before it, because the Actor's identity selects the policy, and the policy supplies the rules. + +1. The gateway identifies the Actor. It reads the Actor's SPIFFE ID from the client certificate on the connection from the Actor, and it verifies this certificate rather than trusting anything in the request. + +2. The gateway fetches the `EgressPolicy` for this Actor. Each Actor has at most one `EgressPolicy` named `default`, and Agent Substrate addresses the policy by the Actor rather than by this name. An Actor with no policy reaches nothing, because a policy is the only thing that allows a destination. + +3. The gateway picks which rule applies to the request. Rules are an unordered set rather than a list, and exactly one rule applies to a given destination name and port regardless of its protocol. When more than one rule matches, the gateway applies two criteria in order, moving to the second only when the first ties. -The gateway identifies the Actor from the client certificate that the connection presents, reading the Actor's SPIFFE ID out of a verified certificate rather than from anything in the request. Nothing running inside the tunnel can set or change that identity, and a request that arrives without one is refused. Two Actors that dial the same address never share a connection pool, so neither inherits the other's identity. + 1. A matching hostname without a wildcard wins over a matching hostname with one. + 2. A matching port number wins over a rule that covers all ports. -The gateway then authorizes the request against the Actor's `EgressPolicy`. An Actor has at most one, named `default`. The policy holds an ordered list of rules, and the gateway takes the **first** rule that matches: only that rule applies, and evaluation stops even when a later rule would also match. **A request that matches no rule is denied, and an Actor with no policy at all gets no outbound connection.** A rule can match only what the gateway can read of the request. + A request that no rule matches is denied, and only the rule that applies contributes a credential. + +The verified identity holds the other two steps together. The Actor's identity is established only by the certificate, so the gateway authorizes every request against the Actor that actually sent it regardless of the headers or hostnames that the request carries. The gateway refuses a connection that does not present a certificate rather than falling back to a default identity. Each Actor also gets its own connection pool, even when two of them dial the same address, so that every connection carries requests from the one Actor that opened it. > [!NOTE] -> Rule order is significant, and a policy holds at most 256 rules. Because the first match wins, a broad rule placed early makes every narrower rule after it unreachable. +> Two rules that tie on the same name and port are rejected when the policy is admitted, so a policy never reaches the gateway with an ambiguous match. A policy holds at most 256 rules. ### Traffic types -The gateway allows or denies a request on one of two things: the `Host` header of the request, or the address the Actor dialed. Which of the two it uses depends on whether the traffic carries a request that the gateway can read. That difference splits egress into three cases worth keeping apart. +A rule's type fixes the kind of connection it covers and the place that the gateway reads the destination hostname from. Three types cover the three cases. -- **Cleartext HTTP**: The gateway reads the `Host` header, so hostname rules apply and a request that matches no rule is refused. -- **TLS**: The gateway terminates the connection with a certificate minted for the destination name, reads the request inside, and matches it against hostname rules exactly as it does cleartext. It then re-originates its own TLS connection to the destination. An allowed request appears in the gateway log with the `http.host` and `http.path` it was matched on. -- **Opaque TCP**: A tunnel that carries no HTTP request gives the gateway nothing to read, so the gateway allows or denies it by address, at its outer hop. Express those destinations as a `cidrs` rule, because a hostname rule has nothing to match against. +- **`http`**: Cleartext HTTP, meaning HTTP/1.1 and the h2c that gRPC uses. The gateway reads the `Host` header and matches it against the rule's hostnames. TLS to those same names is not allowed by this rule. +- **`https`**: HTTPS, which the gateway intercepts. It terminates the connection with a certificate minted for the destination name, reads the request inside, matches it against the rule's hostnames, and re-originates its own TLS connection onward. HTTP/1.1 and HTTP/2 work inside this connection, and HTTP/3 is not supported. +- **`tls_passthrough`**: TLS forwarded to the named servers without decryption. The gateway cannot read the request, so it decides at the TLS ClientHello instead. -Enforcing hostname rules against encrypted traffic has a cost: because the gateway terminates TLS, it holds the cleartext of every agent request for as long as it takes to authorize it. Holding the cleartext also lets the gateway attach credentials on the agent's behalf. For more information, see [Credential injection](#credential-injection). +All three rule types cover TCP connections. Agent Substrate blocks UDP, and it does not support a protocol in which the server sends the first bytes once the connection opens, such as SMTP or SSH. The gateway reads the destination hostname from the bytes that the client sends first, so those protocols leave the gateway nothing to match against. -## Rule matching +Intercepting HTTPS has a cost. Because the gateway terminates TLS, it holds the cleartext of every agent request for the duration of time it takes to authorize this request. Holding the cleartext also lets the gateway attach credentials on the agent's behalf. For more information, see [Credential injection](#credential-injection). -Each rule matches on exactly one kind of destination, and that kind also determines how the gateway dials a connection that the rule allows. +## Rule matching -| Rule | Matches | How the gateway dials a match | -| ---- | ------- | ----------------------------- | -| `hostnames` | The request's hostname, against a list of DNS name patterns. | Resolves the name and dials it by name. | -| `cidrs` | The original destination IP, against a list of IPv4 or IPv6 prefixes. | Dials the original address. | -| `all` | Every destination. | Dials the original address. | +Every rule names the hosts and ports that it covers, regardless of its protocol. -A hostname pattern is a DNS name, optionally with a `*` wildcard replacing the complete leftmost label. The wildcard matches exactly one non-empty label, so `*.example.com` matches `api.example.com` but matches neither `example.com` nor `nested.api.example.com`. No other wildcard syntax is accepted, and a pattern without one matches only the complete name. Patterns must be lowercase, must not carry a trailing dot, and must use the punycode form for an internationalized name. A URL, a bare IP address, and a name with a port are all rejected rather than coerced. +| Field | What it matches | +| ----- | --------------- | +| `hostnames` | A list of DNS name patterns. The rule matches when any pattern matches the destination. | +| `ports` | Either every port, or a list of up to 16 port numbers. | +| `effects` | Not a match condition. See [Credential injection](#credential-injection). | -A CIDR prefix must be canonical, with every bit after the prefix length set to zero, such as `192.0.2.0/24` or `2001:db8::/32`. +A hostname pattern is a DNS name, optionally with a `*` wildcard replacing the complete leftmost label. The wildcard matches exactly one non-empty label, so `*.example.com` matches `api.example.com` but matches neither `example.com` nor `nested.api.example.com`. A `*` wildcard on its own matches every name. No other wildcard syntax is accepted, and a pattern without one matches only the complete name. Patterns must be lowercase, must not carry a trailing dot, and must use the punycode form for an internationalized name. ## Policy generation -kagent derives the allowlist from the AgentTemplate rather than taking it as configuration, so the policy always describes what the agent was actually compiled to need. Each {{< gloss "Revision" >}}revision{{< /gloss >}} records the hosts its agent depends on: +kagent derives the allowlist from the AgentTemplate rather than taking it as configuration, so the policy describes what the agent was compiled to need. Each {{< gloss "Revision" >}}revision{{< /gloss >}} records the hosts that its agent depends on: - The model provider endpoint, from the ModelConfig. - Every RemoteMCPServer, HTTP tool, and SSE tool that the AgentTemplate binds. - Every skill artifact source, such as a git or OCI registry host. - The OpenTelemetry collector, for each telemetry signal that is enabled. -kagent compiles that set into the Actor's policy as it creates the Actor. Each destination that carries a credential takes its own `hostnames` rule, placed first so that the rule holding the credential is the one that matches. The remaining destinations follow as a single `hostnames` rule, and any destination given as a literal IP address becomes one `cidrs` rule at the end. +kagent compiles this host set into the Actor's policy when it creates the Actor. Each destination becomes one rule that covers one hostname on one port, and the scheme decides the rule type. For example, an `https` destination becomes an `https` rule, and a cleartext destination becomes an `http` rule. Keeping exactly one rule per host and port is deliberate; otherwise, an overlapping rule could decide a request and skip the credential that the narrower rule carries. A destination that uses both HTTP and HTTPS on the same host and port is rejected rather than compiled. -> [!IMPORTANT] -> No field adds an arbitrary host to the allowlist. A destination becomes reachable by being named in the AgentTemplate as a model endpoint, an MCP server, an HTTP tool, or a skill source. kagent writes these as hostname rules, which the gateway enforces against encrypted and cleartext requests alike, so the allowlist is the limit of what an agent can reach over HTTP and HTTPS rather than only a record of intent. +The host set is fixed when kagent compiles the policy, so an agent reaches only the hosts that its AgentTemplate declares. The gateway denies every other destination, including a web page that a model decides to fetch partway through a Session. -The collector host is added automatically for agents on every runtime, `byo` included. An image on the `byo` runtime that exports to a backend of its own, through a Harness `spec.env` entry, has no route to that backend, because no field adds its host to the allowlist. For more information, see [Runtime coverage]({{< link path="observability/tracing#runtime-coverage" >}}). + + +An agent also needs a route to the OpenTelemetry collector, or its traces never leave the sandbox and the kagent UI shows nothing for this Session. kagent adds the collector host for agents on every runtime, `byo` included. A `byo` image that exports to a telemetry backend of its own, through a Harness `spec.env` entry, has no route to that backend, because the AgentTemplate never named it. For more information, see [Runtime coverage]({{< link path="observability/tracing#runtime-coverage" >}}). ## Policy changes -The gateway caches each Actor's policy rather than fetching it per request, so a change takes effect on a delay rather than instantly. The cache holds an entry for 10 seconds by default, and that interval is the upper bound on the lag: a policy that is created, updated, or deleted reaches new requests within one interval. Deleting a policy becomes a denial rather than an absence of one. +The gateway caches each Actor's policy rather than fetching it per request, so policy changes are eventually consistent rather than immediate. The cache holds an entry for 10 seconds by default. This interval bounds the lag, so a policy that is created, updated, or deleted reaches new requests within one interval. Deleting a policy produces denials rather than an absence of enforcement because the gateway caches the absence too. -Because a Session pins the revision it was created from, changing an AgentTemplate does not move an existing conversation onto a new allowlist. Create a new Session to pick up a changed set of destinations. +> [!IMPORTANT] +> A Session pins the revision it was created from, so editing an AgentTemplate does not move an existing conversation onto a new allowlist. A Session keeps the destinations that it started with for however long it runs. To pick up a changed set of destinations, create a new Session. ## Diagnose a denied request -A denied request fails without explaining itself: the agent sees a rejected call, and the agent's own error text names neither the destination nor the rule. The gateway records the reason instead. +A denied request fails without explaining itself. The agent sees a rejected call, and its own error text names neither the destination nor the rule that refused it. The gateway records the reason in its log. ```bash kubectl logs -n ate-system deploy/atenet-egress -c agentgateway --tail=100 ``` -A denial appears as a `403` carrying the Actor that made the request, the host it asked for, and why the request failed. +A denial appears as an HTTP `403` alongside the Actor that made the request and the host it asked for. ```console error request ... http.host=kagent-tools.kagent http.status=403 @@ -87,15 +98,26 @@ error request ... http.host=kagent-tools.kagent http.status=403 error="actor egress policy denied: ... \"EgressPolicy not found\"" reason=Authorization ``` -Read `reason=Authorization` as the policy refusing the request, and the `error` text as which check failed. An allowed request is logged the same way, with its status, the `http.host` and `http.path` it was matched on, and the `substrate.connect.authority` address the Actor originally dialed. The log is therefore the quickest way to see what an agent actually reaches. +Read this entry in three parts. `reason=Authorization` means that the policy refused the request rather than the request failing due to some other cause. The `error` text names which check failed, such as a policy that does not exist. The `http.host` names the destination that the agent asked for. Compare this host against the AgentTemplate. + +A `503` means that the gateway could not reach the control plane or a credential provider, which is an installation problem rather than a denied request. + +An allowed request is logged the same way, with its status, the `http.host` and `http.path` it matched on, and the `substrate.connect.authority` address the Actor originally dialed. The log is therefore the quickest way to see what an agent actually reaches. -The compiled policy is not readable through `kubectl` or `kagent`, so treat the AgentTemplate as the source of truth for what an agent is meant to reach. If a destination is missing, it is missing from the AgentTemplate. +The compiled policy is not readable through `kubectl` or `kagent`, so treat the AgentTemplate as the source of truth for what an agent is meant to reach. A destination that is missing from the log is missing from the AgentTemplate. ## Credential injection -An egress rule can declare that the gateway attach a credential to a matching request, so that an agent reaches a protected service without ever holding the secret itself. A `hostnames` rule carries the effect, which names the header to set, an optional prefix such as `Bearer `, and an `ate-secret://` reference that a registered credential provider resolves. +A model provider or an MCP server usually requires an API key or token. The agent sends a placeholder header in place of the real value, and the gateway replaces this header with the credential as the request passes through. The key never enters the agent's environment or its filesystem. -kagent writes these effects for you. Every ModelConfig that holds an `apiKeySecret` becomes a credential binding on the rule for that model's host, as does every RemoteMCPServer whose `headersFrom` reads a Secret. The gateway resolves the reference as the request passes through, so the agent's own environment never holds the key. +kagent sets this up from two resources. Every ModelConfig that holds an `apiKeySecret` becomes a credential on the rule for that model's host, and every RemoteMCPServer whose `headersFrom` reads a Secret becomes a credential on the rule for its host. kagent attaches the credential to the destination's own rule rather than adding a rule for it, and it refuses to compile a credential whose host is not already an allowed destination. + +A credential is declared on a rule as a `replace_headers` entry. The entry names the header to replace, an optional prefix such as `Bearer `, and either a reference that a registered credential provider resolves or a Substrate-issued JWT asserting the Actor's identity. + +> [!IMPORTANT] +> The gateway replaces a header only when the request already carries it. A request that omits the placeholder header is forwarded unchanged, without the credential, rather than being denied. > [!NOTE] -> The gateway injects a credential only on a connection that it terminates, and only when a credential provider is configured for the reference's authority. Where either is missing, the request reaches the destination **without** the credential rather than being denied. Once injection is attempted and fails, the request fails closed: `403` when the provider does not hold the credential or refuses to release it, and `503` when the provider is unreachable. +> Replacement happens only on a connection that the gateway terminates, and only when a credential provider is configured for the reference. If either is missing, the request reaches the destination without the credential. The request fails closed when replacement is attempted and fails. `403` is returned when the provider does not hold the credential or refuses to release it, and `503` is returned when the provider is unreachable. + +Replacement also works on a cleartext `http` rule, where the gateway does not originate TLS to the destination. The credential is then readable by anything on the path between the gateway and the destination unless something outside the gateway encrypts that segment. Prefer an `https` destination for any host that takes a credential. diff --git a/docs-site/content/kagent/1.x/about/substrate-runtime/sandboxing.md b/docs-site/content/kagent/1.x/about/substrate-runtime/sandboxing.md index 1ae4ea29..90cbc050 100644 --- a/docs-site/content/kagent/1.x/about/substrate-runtime/sandboxing.md +++ b/docs-site/content/kagent/1.x/about/substrate-runtime/sandboxing.md @@ -39,7 +39,7 @@ A **sandbox class** is the sandbox runtime family that a Worker uses. {{< gloss A {{< gloss "WorkerPool" >}}WorkerPool{{< /gloss >}}'s `sandboxClass` field sets the sandbox class for its Workers, and defaults to `gvisor`. That value does more than name a runtime. Agent Substrate builds the pool's Worker pods to match it, mounting the virtualization devices that a micro-VM needs and placing those pods on nodes that provide them. > [!NOTE] -> kagent copies a pool's sandbox class onto every {{< gloss "ActorTemplate" >}}ActorTemplate{{< /gloss >}} that it generates for that pool. Each template then names the SandboxConfig that serves its class. A `gvisor` template names `gvisor-default`, and a `microvm` template names `microvm`. +> kagent copies a pool's sandbox class onto every {{< gloss "ActorTemplate" >}}ActorTemplate{{< /gloss >}} that it generates for this pool. Each template then names the SandboxConfig that serves its class. A `gvisor` template names `gvisor-default`, and a `microvm` template names `microvm`. ## Sandbox configuration @@ -47,7 +47,7 @@ A **SandboxConfig** is a cluster-scoped resource that describes the runtime file Every ActorTemplate must name a SandboxConfig, and Agent Substrate supplies no name of its own. A template cannot be prepared until the configuration that it names exists in the cluster. Many ActorTemplates can share one SandboxConfig. The runtime version is then pinned in a single place, rather than repeated on every template. -A default installation creates one SandboxConfig, named `gvisor-default`. kagent writes that name onto every ActorTemplate that it generates, so each template carries it explicitly. The configuration looks like the following: +A default installation creates one SandboxConfig, named `gvisor-default`. kagent writes this name onto every ActorTemplate that it generates, so each template carries it explicitly. The configuration looks like the following: ```yaml apiVersion: ate.dev/v1alpha1 @@ -87,19 +87,19 @@ The sandbox draws a boundary in three places. Every Actor is addressed by its {{< gloss "Atespace" >}}atespace{{< /gloss >}} and name, at `..actors.resources.substrate.ate.dev`. Reaching it involves several hops, and each one keeps a sandboxed Actor addressable without exposing the Worker that it happens to be running on. -1. Agent Substrate runs its own Domain Name System (DNS) service that answers queries for that address pattern with the address of the router, rather than any individual Worker. -2. The router reads the Actor name and atespace from the request, asks the Agent Substrate API to resume that Actor and report which Worker it is now assigned to, then selects that Worker as the destination. -3. The router connects to a listener on the Worker over mutual Transport Layer Security (mTLS). The listener validates that the caller is the router, and forwards traffic only to the Actor currently assigned to that Worker. +1. Agent Substrate runs its own Domain Name System (DNS) service that answers queries for this address pattern with the address of the router, rather than any individual Worker. +2. The router reads the Actor name and atespace from the request, asks the Agent Substrate API to resume this Actor and report which Worker it is now assigned to, then selects this Worker as the destination. +3. The router connects to a listener on the Worker over mutual Transport Layer Security (mTLS). The listener validates that the caller is the router, and forwards traffic only to the Actor currently assigned to this Worker. Because the router resolves the Worker assignment on every request, an Actor keeps a stable address across suspends, resumes, and moves between Workers. -A kagent user never uses that address. A message sent to a {{< gloss "Session" >}}Session{{< /gloss >}} through the kagent API or the web console reaches the kagent controller, which resolves the Session to its Actor and passes the request to Agent Substrate. Every hop after that stays inside the runtime. +A kagent user never uses this address. A message sent to a {{< gloss "Session" >}}Session{{< /gloss >}} through the kagent API or the web console reaches the kagent controller, which resolves the Session to its Actor and passes the request to Agent Substrate. Every hop after that stays inside the runtime. Traffic in the other direction leaves through a separate egress gateway rather than going straight out from the Worker. Routing Actor egress through one gateway provides a single place to apply outbound controls. ## Default network posture -Agent Substrate creates a Kubernetes NetworkPolicy for each WorkerPool, selecting that pool's Worker pods. The policy restricts **ingress** to the Agent Substrate router alone. No other pod in the cluster can open a connection to a Worker, so an Actor is not reachable by anything that bypasses the routing path. +Agent Substrate creates a Kubernetes NetworkPolicy for each WorkerPool, selecting this pool's Worker pods. The policy restricts **ingress** to the Agent Substrate router alone. No other pod in the cluster can open a connection to a Worker, so an Actor is not reachable by anything that bypasses the routing path. That policy governs inbound traffic only. Outbound traffic is governed separately by an egress policy that the gateway enforces on every connection that an Actor opens. That policy is default-deny. An Actor reaches a destination only when a rule allows it, and the gateway denies the connection when no rule does. diff --git a/docs-site/content/kagent/1.x/about/substrate-runtime/suspend-and-resume.md b/docs-site/content/kagent/1.x/about/substrate-runtime/suspend-and-resume.md index 22757487..837ef171 100644 --- a/docs-site/content/kagent/1.x/about/substrate-runtime/suspend-and-resume.md +++ b/docs-site/content/kagent/1.x/about/substrate-runtime/suspend-and-resume.md @@ -7,7 +7,7 @@ aliases: - /kagent/1.x/substrate-runtime/suspend-and-resume/ --- -An agent spends most of its life waiting. It waits on a person to reply, and it waits on a large language model (LLM) to answer. [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}) runs each agent inside an **Actor**, the unit that it suspends and resumes. Rather than hold compute open through that waiting, Agent Substrate writes the idle Actor to a {{< gloss "Snapshot" >}}snapshot{{< /gloss >}} and releases the capacity that the Actor held on its {{< gloss "Worker" >}}Worker{{< /gloss >}}. The next message restores it. The {{< gloss "Session" >}}Session{{< /gloss >}} stays ready the whole time, so reclaiming that capacity costs the user nothing. This page explains what a snapshot captures, when kagent suspends an Actor, and what happens when a suspended Actor is addressed again. +An agent spends most of its life waiting. It waits on a person to reply, and it waits on a large language model (LLM) to answer. [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}) runs each agent inside an **Actor**, the unit that it suspends and resumes. Rather than hold compute open through that waiting, Agent Substrate writes the idle Actor to a {{< gloss "Snapshot" >}}snapshot{{< /gloss >}} and releases the capacity that the Actor held on its {{< gloss "Worker" >}}Worker{{< /gloss >}}. The next message restores it. The {{< gloss "Session" >}}Session{{< /gloss >}} stays ready the whole time, so reclaiming this capacity costs the user nothing. This page explains what a snapshot captures, when kagent suspends an Actor, and what happens when a suspended Actor is addressed again. ## Actor lifecycle operations @@ -22,7 +22,7 @@ The following diagram traces an Actor through those operations, and shows the fu {{< reuse "kagent-docs/snippets/snapshot-cycle-diagram.md" >}} -An Actor reports its position in that cycle through its state, which is one of `RESUMING`, `RUNNING`, `SUSPENDING`, `SUSPENDED`, `PAUSING`, `PAUSED`, `CRASHED`, or `DELETING`. Only a suspended Actor can be deleted. +An Actor reports its position in this cycle through its state, which is one of `RESUMING`, `RUNNING`, `SUSPENDING`, `SUSPENDED`, `PAUSING`, `PAUSED`, `CRASHED`, or `DELETING`. Only a suspended Actor can be deleted. > [!NOTE] > Resume restores an Actor onto whichever Worker in the pool has room, which is not necessarily the Worker that the Actor ran on before. Because a suspended Actor occupies no Worker, a {{< gloss "WorkerPool" >}}WorkerPool{{< /gloss >}} can serve far more Sessions than it could if every Actor stayed resident. @@ -34,22 +34,20 @@ An {{< gloss "ActorTemplate" >}}ActorTemplate{{< /gloss >}}'s snapshot configura - **`Full`**: Captures process memory, the root filesystem changes layered on top of the container image, and any attached durable volumes. A `Full` snapshot holds everything that is needed to resume the Actor hot, with its in-memory state intact. - **`Data`**: Captures only the contents of attached durable volumes. Process memory and the rest of the root filesystem are discarded, which makes the snapshot much cheaper to write and store. -kagent sets these scopes automatically on every ActorTemplate that it generates. A pause captures `Full`, a commit captures `Data`, a resume cold-boots, and each Actor gets a single durable volume named `data`. The one snapshot setting that you author is the storage location, which you set on the {{< gloss "Harness" >}}Harness{{< /gloss >}}. +kagent sets the scope automatically on every ActorTemplate that it generates, and it sets `Full` so that process memory survives both a pause and a suspend. Each Actor also gets a single durable volume named `data`. The one snapshot setting that you author is the storage location, which you set on the {{< gloss "Harness" >}}Harness{{< /gloss >}}. -A scope is set per trigger. The `onPause` setting selects what a pause captures on the node, and `onCommit` selects what a suspend uploads to snapshot storage. What `onCommit` captures must be a subset of what `onPause` captures. +One scope governs both operations. The ActorTemplate's `onCommit` setting selects what a suspend uploads to snapshot storage, and a pause captures this same scope on the node. ### Durable volumes and resume A **DurableDir volume** is the per-Actor application data surface. Its contents are preserved by the `Data` scope, so they survive a suspend and resume cycle independently of process memory. How many volumes an ActorTemplate can declare depends on its sandbox class. A `microvm` template can declare several, because they are subdirectories of a single shared filesystem. A `gvisor` template is limited to one, until {{< gloss "gVisor" >}}gVisor{{< /gloss >}} accepts more than a single durable mount. -When an Actor resumes from a `Data`-scope snapshot, the ActorTemplate's `onResume.fromData` setting decides where the rest of the guest state comes from. The default is `ColdBoot`, which starts the containers fresh from the container image with the durable volume contents restored over them. - ## Golden and per-Actor snapshots Two kinds of snapshot serve different purposes, and both appear in a normal installation. -- The **golden snapshot** is shared by every Actor of one ActorTemplate. Agent Substrate captures it once by booting the workload temporarily when the template is created. Every new Actor of that template starts from this snapshot rather than from a cold start. kagent waits for the golden snapshot before an {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}} becomes ready, and the controller reports `waiting for the ActorTemplate golden snapshot` while the capture runs. -- The **last snapshot** belongs to one Actor. Agent Substrate writes it on every suspend and restores that same Actor from it on the next resume. It holds the files and memory that this one conversation built up, which is how the conversation continues from where it stopped. +- The **golden snapshot** is shared by every Actor of one ActorTemplate. Agent Substrate captures it once by booting the workload temporarily when the template is created. Every new Actor of this template starts from this snapshot rather than from a cold start. kagent waits for the golden snapshot before an {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}} becomes ready, and the controller reports `waiting for the ActorTemplate golden snapshot` while the capture runs. +- The **last snapshot** belongs to one Actor. Agent Substrate writes it on every suspend and restores this same Actor from it on the next resume. It holds the files and memory that this one conversation built up, so the conversation continues from where it stopped. Snapshots are persisted to object storage so that Actor state is durable and portable across the cluster. Agent Substrate writes snapshots to Google Cloud Storage or to Amazon Simple Storage Service (S3). The S3 client also reaches S3-compatible stores that accept a custom endpoint and path-style addressing. For the settings that select and configure each client, see [Configure snapshot storage]({{< link path="operations/tune-agent-substrate#configure-snapshot-storage" >}}). A {{< gloss "Harness" >}}Harness{{< /gloss >}} names the location for its Actors' snapshots in its `substrate.snapshotPolicy` section. @@ -89,6 +87,6 @@ Because kagent suspends at every turn boundary, every message to a Session pays A **checkpoint** marks one point in a conversation and keeps it. You can create a checkpoint when a Session reaches a state worth returning to, such as a long investigation that is only partway through. You can then start a second Session from that point, which is known as a {{< gloss "Fork" >}}fork{{< /gloss >}}. The fork continues the conversation from where the checkpoint marked it, new turns append only to the fork, and the fork gets its own conversation and task identifiers rather than inheriting the source's. To create a checkpoint and fork a Session from it, follow the [Agent Substrate example]({{< link path="examples/agent-substrate" >}}). -Checkpoint a Session only at a turn boundary, because the turn boundary is what gets captured. The request names the terminal task to pin, and it fails when a newer turn has already started or while that turn's snapshot is still being written. +Checkpoint a Session only at a turn boundary, because a checkpoint captures this boundary. The request names the terminal task to pin, and it fails when a newer turn has already started or while this turn's snapshot is still being written. Underneath, a checkpoint pins a snapshot that would otherwise be discarded. Agent Substrate treats the snapshot written on each suspend as transient and collects it once a newer one supersedes it. Creating a checkpoint attaches a {{< gloss "Tag" >}}Tag{{< /gloss >}} to the snapshot that the Session most recently suspended to, and Agent Substrate keeps any snapshot that a tag still names. Deleting the checkpoint removes the tag and releases the snapshot. diff --git a/docs-site/content/kagent/1.x/operations/tune-agent-substrate.md b/docs-site/content/kagent/1.x/operations/tune-agent-substrate.md index b9a24626..886d7061 100644 --- a/docs-site/content/kagent/1.x/operations/tune-agent-substrate.md +++ b/docs-site/content/kagent/1.x/operations/tune-agent-substrate.md @@ -103,7 +103,7 @@ substrateWorkerPool: ### Recognize an undersized pool -Agent Substrate reports pool capacity through its metrics rather than through an error that reaches the caller. Each Worker publishes a state of `idle`, `partial`, `at_capacity`, or `unschedulable`, which together show how much of a pool is in use, and a placement that finds no suitable Worker records a `no_capacity` outcome. Agent Substrate treats that outcome as a normal result rather than a failure, so it carries no error type and raises no alert on its own. Watch the share of Workers at `at_capacity` and the rate of `no_capacity` placements to tell a pool that is merely busy from one that is too small. +Agent Substrate reports pool capacity through its metrics rather than through an error that reaches the caller. Each Worker publishes a state of `idle`, `partial`, `at_capacity`, or `unschedulable`, which together show how much of a pool is in use, and a placement that finds no suitable Worker records a `no_capacity` outcome. Agent Substrate treats this outcome as a normal result rather than a failure, so it carries no error type and raises no alert on its own. Watch the share of Workers at `at_capacity` and the rate of `no_capacity` placements to tell a pool that is merely busy from one that is too small. None of those signals reach the caller. A turn that cannot get a Worker waits for one, and a turn that waits too long fails with a generic timeout that names neither capacity nor the pool. @@ -185,7 +185,7 @@ Because the two places are configured independently, confirm the result rather t A pool's sandbox class decides which sandbox runtime its Workers provide, and kagent constrains the choice more tightly than Agent Substrate does. -Agent Substrate supports the `gvisor` and `microvm` classes, as explained in [Sandboxing]({{< link path="about/substrate-runtime/sandboxing" >}}). kagent compiles every ActorTemplate to the class that its pool selected and to the matching SandboxConfig: a pool on `gvisor` (or no `sandboxClass`, which defaults to `gvisor`) gets the `gvisor` class and a SandboxConfig named `gvisor-default`, while a pool on `microvm` gets the `microvm` class and a SandboxConfig named `microvm`. Placement never relaxes the class constraint: a Worker in a pool whose class does not match an ActorTemplate's class reports `RevisionInvalid` with `unsupported sandbox class ""`, and placement of that Actor fails with `WorkerPoolNotFound`. +Agent Substrate supports the `gvisor` and `microvm` classes, as explained in [Sandboxing]({{< link path="about/substrate-runtime/sandboxing" >}}). kagent compiles every ActorTemplate to the class that its pool selected and to the matching SandboxConfig: a pool on `gvisor` (or no `sandboxClass`, which defaults to `gvisor`) gets the `gvisor` class and a SandboxConfig named `gvisor-default`, while a pool on `microvm` gets the `microvm` class and a SandboxConfig named `microvm`. Placement never relaxes the class constraint: a Worker in a pool whose class does not match an ActorTemplate's class reports `RevisionInvalid` with `unsupported sandbox class ""`, and placement of this Actor fails with `WorkerPoolNotFound`. Leave a pool that backs kagent Harnesses on the class you intend to run, and keep the pool's image on the matching Worker build: `ateom-gvisor` for a pool on `gvisor`, `ateom-microvm` for a pool on `microvm`. From 12107579ad3bc60c42817826f67de73001ca86de Mon Sep 17 00:00:00 2001 From: Rachael Graham Date: Mon, 12 Oct 2026 14:02:44 -0500 Subject: [PATCH 05/10] Clarify production operations guidance Expand operational docs on database and controller availability, Worker node lifecycle risks, idle Session expiry, and provider key rotation. Also refine proxy guidance and correct the Session expiry link. Signed-off-by: Rachael Graham --- .../content/kagent/1.x/about/core-concepts.md | 2 +- .../operations/operational-considerations.md | 92 +++++++++++-------- 2 files changed, 57 insertions(+), 37 deletions(-) diff --git a/docs-site/content/kagent/1.x/about/core-concepts.md b/docs-site/content/kagent/1.x/about/core-concepts.md index ae0dd370..cda529b7 100644 --- a/docs-site/content/kagent/1.x/about/core-concepts.md +++ b/docs-site/content/kagent/1.x/about/core-concepts.md @@ -135,7 +135,7 @@ kagent agent session create --agent assistant -n kagent After it is created, a Session talks to callers over the {{< gloss "A2A" >}}A2A{{< /gloss >}} (Agent-to-Agent) protocol, through kagent's A2A gateway. Callers address the Agent rather than the Session. The HTTP endpoint is `/agents/{namespace}/{name}` and gRPC carries the same `namespace/name` in the standard A2A `tenant` field. The Session's ID is the A2A `contextId`, so a message that carries no context identifier starts a new conversation, and a message that repeats one continues that conversation. -A Session that records no task activity for seven days is deleted by an expiration worker. The `controller.sessionIdleTTL` Helm value sets that window. A value of `0` turns the expiration worker off. For more information about what the deletion retains, see [Expire idle conversations]({{< link path="operations/operational-considerations#expire-idle-conversations" >}}). +A Session that records no task activity for seven days is deleted by an expiration worker. The `controller.sessionIdleTTL` Helm value sets that window. A value of `0` turns the expiration worker off. For more information about what the deletion retains, see [Expire idle conversations]({{< link path="operations/operational-considerations#expire-idle-sessions" >}}). ## Actor diff --git a/docs-site/content/kagent/1.x/operations/operational-considerations.md b/docs-site/content/kagent/1.x/operations/operational-considerations.md index b64da518..f50f43de 100644 --- a/docs-site/content/kagent/1.x/operations/operational-considerations.md +++ b/docs-site/content/kagent/1.x/operations/operational-considerations.md @@ -1,15 +1,15 @@ --- title: Operational considerations -description: Replace the evaluation defaults for the database, controller replicas, and the node pools that host Substrate Workers before you run kagent in production. +description: Tune the database, controller replicas, and the node pools that host Substrate Workers before you run kagent in production. weight: 10 author: kagent.dev --- -A default {{< reuse "kagent-docs/snippets/name-product.md" >}} installation is built for evaluation. It runs one controller replica against a bundled database, and places no constraints on the nodes that run agents. Production changes each of those. +A default {{< reuse "kagent-docs/snippets/name-product.md" >}} installation runs one controller replica against a bundled database, and schedules agents onto any node in the cluster. Each of those defaults works for a trial installation, but all three need a decision before you run {{< reuse "kagent-docs/snippets/name-product.md" >}} in production. This page covers which database to run, how many controller replicas to keep, and how to treat the nodes that host Workers. ## Choose a database -kagent stores conversations, {{< gloss "Session" >}}Sessions{{< /gloss >}}, and compiled {{< gloss "Revision" >}}revisions{{< /gloss >}} in PostgreSQL. A bundled instance ships with the chart so that an evaluation needs no external prerequisites, and production deployments supply their own. +kagent stores conversations, {{< gloss "Session" >}}Sessions{{< /gloss >}}, and compiled {{< gloss "Revision" >}}revisions{{< /gloss >}} in PostgreSQL. A bundled instance ships with the chart and runs by default, so an installation needs no external database to start. Run your own PostgreSQL instance for production, because the bundled one is a single pod backed by one PersistentVolumeClaim (PVC). Two independent settings determine what runs and what the controller talks to: @@ -75,7 +75,7 @@ For production, run PostgreSQL outside the cluster's lifecycle so that a kagent --values kagent.yaml ``` -## Run multiple controller replica +## Run multiple controller replicas To ensure that a controller failure does not stop reconciliation, set `controller.replicas` to a number higher than `1`. @@ -84,23 +84,34 @@ controller: replicas: 3 ``` -Leader election keeps the replicas from conflicting. One replica holds a Kubernetes lease and performs reconciliation, garbage collection, and scheduled runs; the other replicas stay ready and take over when the leader's lease expires. +Leader election keeps the replicas from conflicting. One replica holds a Kubernetes lease and performs reconciliation, garbage collection, and scheduled runs. The other replicas stay ready and take over when the leader's lease expires. -> [!NOTE] -> Leader election is always on, including at a single replica, because a rolling update briefly runs two controllers at once. The chart grants the lease permissions unconditionally and exposes no setting to turn election off. `KAGENT_LEADER_ELECT=false` remains available for local testing. +> [!IMPORTANT] +> If an installation runs on the bundled database, extra controller replicas do not make the installation highly available. The bundled PostgreSQL instance is a single pod backed by one PVC, so losing it stops every replica at once. Run an external PostgreSQL instance before counting controller replicas toward availability. + +## Node operations + +A {{< gloss "Worker" >}}Worker{{< /gloss >}} is an ordinary Kubernetes pod, so every node event reaches the {{< gloss "Actor" >}}Actors{{< /gloss >}} that this Worker hosts. What the event costs depends on what each Actor was doing when the node went away. + +* A suspended Actor costs nothing. Its state is already written to snapshot storage, and it occupies no Worker while it waits, so the next message resumes it onto whichever Worker in the pool has availability. +* An awake Actor does not survive. Its conversation lives in the memory of its sandbox on the Worker, and Agent Substrate writes nothing to snapshot storage until this Actor suspends. -PostgreSQL supports multiple controller replicas without further configuration. The bundled instance is still a single pod backed by one PVC, so an installation that runs several controllers for availability, against a bundled database, has only moved the single point of failure. +A lost Actor moves to `ACTOR_STATE_CRASHED`, and Agent Substrate refuses `resume` and `suspend` on an Actor in this state. Through kagent, deleting the Actor and starting a new {{< gloss "Session" >}}Session{{< /gloss >}} is the only available option, so the conversation ends wherever the node ends. -## Reserve node pools for Workers + -{{< gloss "Worker" >}}Workers{{< /gloss >}} hold running {{< gloss "Actor" >}}Actors{{< /gloss >}}. An Actor that loses its node before it suspends loses its conversation, so the node pools that run Workers need stricter rules than the rest of the cluster. +Many Actors on a node are suspended at any moment, so losing the node crashes only the ones that are mid-turn. Those Actors still lose their conversations, and a node pool that runs Workers therefore needs stricter rules than the rest of the cluster. A Worker pod that Kubernetes deletes gives its Actors time to suspend. Node auto-upgrade, a spot reclaim, and a pool scale-down do not. -> [!WARNING] -> Turn node auto-upgrade off on every node pool that runs Workers, and do not use spot or preemptible nodes for them. An Actor that is still awake when its node goes away moves to `ACTOR_STATE_CRASHED`. That state is terminal. Agent Substrate refuses both `resume` and `suspend` on a crashed Actor, offers no recovery verb, and cannot start it from the {{< gloss "Snapshot" >}}snapshot{{< /gloss >}} that the Actor still holds. Deleting the Actor and creating a new one is the only available option, and the conversation does not survive. +### Graceful pod deletion -A graceful pod deletion is safe. Kubernetes forwards `SIGTERM` into the Actor's containers, the Worker drains for up to 30 minutes inside a pod termination grace period of 3600 seconds, and an Actor that suspends inside that window keeps its state and stays resumable. Rolling a `workerImage` change through a pool therefore finishes any in-flight turns rather than cutting them off. +Deleting a Worker pod is safe. Kubernetes forwards `SIGTERM` into the Actor's containers, the Worker drains for up to 30 minutes inside a pod termination grace period of 3600 seconds, and an Actor that suspends inside this window keeps its state and stays resumable. Rolling a `workerImage` change through a pool therefore finishes any in-flight turns rather than cutting them off. -A reclaimed node is not a graceful deletion, and node auto-upgrade is the most likely way to hit one. Google Kubernetes Engine (GKE) enables auto-upgrade by default and runs it on Google's maintenance schedule, not yours. Disable it on every pool that runs Workers. +### Node auto-upgrade + +A reclaimed node is not a graceful deletion, and node auto-upgrade is the most likely way to hit one. Google Kubernetes Engine (GKE) enables auto-upgrade by default and runs it on Google's maintenance schedule, not yours. + +> [!IMPORTANT] +> Turn node auto-upgrade off on every node pool that runs Workers. ```bash gcloud container node-pools update "${NODE_POOL}" \ @@ -108,28 +119,39 @@ gcloud container node-pools update "${NODE_POOL}" \ --no-enable-autoupgrade ``` -Scaling a serving WorkerPool down removes pods without suspending the Actors on them, so it strands conversations exactly as a reclaimed node does. For pool sizing and the rest of the Substrate runtime settings, see [Tune Agent Substrate]({{< link path="operations/tune-agent-substrate" >}}). +### Spot and preemptible nodes -## Expire idle conversations +A provider reclaims spot and preemptible nodes on its own schedule, and each reclaim crashes every Actor that was awake on the node. A pool on reclaimable capacity therefore trades a lower bill for conversations that end without warning. Use these nodes only where the agents on this pool can afford to lose a turn. -A conversation that nobody returns to still holds a row in your database and a pinned runtime revision. The kagent controller deletes idle {{< gloss "Session" >}}Sessions{{< /gloss >}} on a timer so that neither accumulates without a bound. +### WorkerPool scale-down -A leader-only worker sweeps once a minute and deletes every Session whose idle clock has run out, through the same deletion workflow that a client delete uses. +When you scale a serving WorkerPool down, it removes pods without suspending the Actors on them. This strands Sessions in the same way that a reclaimed node does. For pool sizing and the rest of the Substrate runtime settings, see [Tune Agent Substrate]({{< link path="operations/tune-agent-substrate" >}}). -| Value | Default | Description | -| ----- | ------- | ----------- | -| `controller.sessionIdleTTL` | `168h` | How long a Session can be idle before the worker deletes it, as a Go duration. `0` turns the worker off, retries included. A negative value is rejected. | +## Expire idle Sessions + +Every {{< gloss "Session" >}}Session{{< /gloss >}} holds database records and pins a runtime revision for as long as it exists, including a Session that is never opened again. The kagent controller deletes idle Sessions on a timer. Deleting idle sessions bounds the records that a kagent installation stores as well as the revisions that it holds open. + +A leader-only worker sweeps Sessions once a minute and deletes every Session that has been idle past the limit, using the same deletion workflow that a client delete uses. + +To choose how long a Session can be idle before the worker deletes it, set `controller.sessionIdleTTL` in your {{< reuse "kagent-docs/snippets/name-product.md" >}} Helm values. The value is a Go duration, and it defaults to `168h`, or seven days. Setting it to `0` turns the worker off, retries included, and a negative value is rejected. + +```yaml +controller: + sessionIdleTTL: "720h" +``` -The `168h` default is seven days. There is no per-agent override and no maximum, so this one value governs every conversation in the installation. +No per-agent override exists, so this one value governs every Session in the installation. No maximum exists either, so a longer value is accepted. -Idle time runs from whichever is later: the Session's creation, or its most recently stored A2A event. Reads, renames, lifecycle calls, and retried writes do not reset the clock, so a conversation that is only ever listed still expires. A {{< gloss "Fork" >}}fork{{< /gloss >}} keeps its source's event timestamps. Its own creation time gives it a full lifetime, so it does not inherit an almost-expired one. +Idle time runs from whichever is later: the Session's creation, or its most recently stored A2A event. Reads, renames, lifecycle calls, and retried writes do not reset it, so a Session that is only ever listed still expires. A {{< gloss "Fork" >}}fork{{< /gloss >}} keeps its source's event timestamps. Its own creation time gives it a full lifetime, so it does not inherit an almost-expired one. Work in progress is never deleted. A Session with a running task survives past the interval, and so does one with a turn waiting at `INPUT_REQUIRED` or `AUTH_REQUIRED`. A pending lifecycle operation or a checkpoint being captured also holds a Session open. > [!IMPORTANT] -> **Expiry bounds conversations, not history.** Deleting an idle Session removes the Session, its shares, its runtime row, and its creation receipt, and `GetSession` then returns not-found. The A2A context, its tasks, its event history, and any explicit {{< gloss "Checkpoint" >}}checkpoints{{< /gloss >}} stay in PostgreSQL, and a retained checkpoint can still be forked after the Session that it was taken on has expired. Sizing a database for a long-running installation means planning for that audit history separately, because no setting on this page bounds it. +> **Expiry bounds Sessions, not history.** Deleting an idle Session removes the Session, its shares, its runtime row, and its creation receipt, so `GetSession` returns not-found afterward. The A2A context, its tasks, its event history, and any explicit {{< gloss "Checkpoint" >}}checkpoints{{< /gloss >}} stay in PostgreSQL, and a retained checkpoint can still be forked after the Session it was taken on has expired. kagent has no setting that bounds this retained history, so size a database for a long-running installation with the audit history planned separately. + +An expired Session releases its creation request ID, unlike an explicit client deletion, which keeps a tombstone. The same caller can reuse that request ID to start a fresh Session. -Unlike an explicit client deletion, which keeps a tombstone, an expired Session releases its creation request ID. The same caller can reuse that request ID to start a fresh conversation. + To watch the sweep, scrape `kagent_session_expired_total`, which counts the Sessions that it removed. Each removal also logs `expired idle session` at debug level with the Session's ID and how long it had been idle. @@ -145,11 +167,18 @@ This behavior differs from kagent 0.x, where an agent ran as a Deployment and a ### Rotate the model provider API key -How you rotate the key depends on who owns the Secret. When you set `providers..apiKey` in your Helm values, the kagent chart creates the Secret and Helm owns it. Rotate the key through Helm, not by editing the Secret, because a direct edit does not last. With Helm 3, the next `helm upgrade --reuse-values` writes the previous key back. With Helm 4, which applies changes on the server side, the next upgrade fails with a field ownership conflict on the Secret. +To rotate the model provider API key, update the Secret that holds the key. Then you can restart the Agent Substrate egress gateway so that running Sessions use the new value. + +Manage the Secret yourself in a production installation. You then rotate the key at its source, and kagent reads the current value. 1. Update the key. {{< tabs >}} + {{% tab name="You manage the Secret" %}} + When `providers..apiKey` is empty and `apiKeySecretRef` names a Secret that you create, such as one that an ExternalSecret syncs, the chart does not render the Secret. Update the Secret wherever you manage it. + {{% /tab %}} {{% tab name="Helm manages the Secret" %}} + When you set `providers..apiKey` in your Helm values, the chart creates the Secret and Helm owns it. Rotate the key through Helm rather than by editing the Secret, because a direct edit does not last. With Helm 3, the next `helm upgrade --reuse-values` writes the previous key back. With Helm 4, which applies changes on the server side, the next upgrade fails with a field ownership conflict on the Secret. + Upgrade the kagent release with the new key. The following example uses OpenAI. For another provider, replace `openAI` with the provider's key in the `providers` values. ```bash helm upgrade kagent \ @@ -160,9 +189,6 @@ How you rotate the key depends on who owns the Secret. When you set `providers.< --set-string providers.openAI.apiKey="${NEW_OPENAI_API_KEY}" ``` {{% /tab %}} - {{% tab name="You manage the Secret" %}} - When `providers..apiKey` is empty and `apiKeySecretRef` names a Secret that you create, such as one that an ExternalSecret syncs, the chart does not render the Secret. Update the Secret wherever you manage it. - {{% /tab %}} {{< /tabs >}} 2. Restart the Agent Substrate egress gateway. The egress gateway can keep sending the previous key after the Secret changes, and a restart makes it use the new key right away. @@ -172,7 +198,7 @@ How you rotate the key depends on who owns the Secret. When you set `providers.< ## Route agent traffic through a proxy -When agents and MCP servers sit behind an API gateway or proxy, point kagent at the proxy endpoint so that agent-to-agent and agent-to-MCP traffic follows the same path as the rest of your cluster's egress. +A proxy puts agent-to-agent and agent-to-MCP calls on the same egress path as the rest of your cluster. Your existing inspection, logging, and policy then cover them too. When agents and MCP servers sit behind an API gateway or proxy, point kagent at the proxy endpoint. ```yaml proxy: @@ -185,12 +211,6 @@ The controller rewrites internally built Kubernetes URLs to the proxy and sets t The controller serves a Prometheus `/metrics` endpoint, turned off by default. To enable the endpoint, grant Prometheus access to it, and review the metrics that the controller reports, see [Metrics]({{< link path="observability/metrics" >}}). -## Isolate what an agent can do - -kagent 0.x ran agents as Deployments and relied on a Kubernetes `securityContext` to constrain them. In 1.0, every agent runs as an Actor inside a gVisor sandbox, and the sandbox provides process, network, and filesystem isolation without per-agent security context configuration. - -For what the sandbox blocks, how to configure egress, and how to select a sandbox class, see [Sandboxing]({{< link path="about/substrate-runtime/sandboxing" >}}). - ## Next steps {{< cards >}} From ed0c69515458e3fa6a36d00fcde97e1d7a8feb77 Mon Sep 17 00:00:00 2001 From: Rachael Graham Date: Mon, 12 Oct 2026 14:41:10 -0500 Subject: [PATCH 06/10] Update tune-agent-substrate.md Signed-off-by: Rachael Graham --- .../1.x/operations/tune-agent-substrate.md | 163 +++++++++++++----- 1 file changed, 116 insertions(+), 47 deletions(-) diff --git a/docs-site/content/kagent/1.x/operations/tune-agent-substrate.md b/docs-site/content/kagent/1.x/operations/tune-agent-substrate.md index 886d7061..b07b46a4 100644 --- a/docs-site/content/kagent/1.x/operations/tune-agent-substrate.md +++ b/docs-site/content/kagent/1.x/operations/tune-agent-substrate.md @@ -5,9 +5,9 @@ weight: 20 author: kagent.dev --- -{{< reuse "kagent-docs/snippets/name-product.md" >}} runs every agent on [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}), and a fresh installation is deliberately small: one {{< gloss "WorkerPool" >}}WorkerPool{{< /gloss >}} holding a single Worker, snapshots in whichever object storage the Agent Substrate installation was given, and the `gvisor` sandbox class. +{{< reuse "kagent-docs/snippets/name-product.md" >}} runs every agent on [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}). A default installation creates one {{< gloss "WorkerPool" >}}WorkerPool{{< /gloss >}} named `kagent-default`, holding a single Worker on the `gvisor` sandbox class, and it writes snapshots to the object store that the Agent Substrate chart bundles. -When preparing for real traffic to your agents, you can size the pool and check where snapshots land. The sandbox class a pool backs can be `gvisor` or `microvm`, as long as the pool's Worker build and SandboxConfig match the class, as described in [Keep pools on the sandbox class they serve](#keep-pools-on-the-sandbox-class-they-serve). +This single Worker reports capacity for 1000 Actors, so a default pool serves many concurrent {{< gloss "Session" >}}Sessions{{< /gloss >}} before it runs out of slots. Three settings require tuning before the installation can carry real traffic: how many Workers to run, where snapshots are stored durably, and what resource requests the Worker pods carry. This page covers your options for tuning each setting, and steps for moving a pool to the `microvm` sandbox class. ## Before you begin @@ -135,13 +135,68 @@ substrateWorkerPool: The same `template` block also accepts `nodeSelector`, `tolerations`, `priorityClassName`, and `nodeAffinity`, so you can keep Workers on nodes that you set aside for them. + + ## Configure snapshot storage -Snapshot storage is configured in two independent places, and a mismatch between them breaks resume rather than install. +Agent Substrate writes every suspended Actor to an object store, so the bucket holds the conversation state that a resume restores. Two places decide where a snapshot lands: + +- In the Agent Substrate Helm values, `atelet.storageBackend` selects the storage client and `rustfs.enabled` decides whether the chart supplies credentials for it. These two values cover both the components that write snapshots and the ones that manage them. +- In each {{< gloss "Harness" >}}Harness{{< /gloss >}}, `spec.substrate.snapshotPolicy.location`, names the bucket and prefix that its Actors write under. The field is required, and it must point into a bucket that the selected client can reach. + +### The bundled object store + +Because `rustfs.enabled` defaults to `true`, the Agent Substrate chart installs an in-cluster object store and points the `atelet` DaemonSet and the `ate-api-server` Deployment at it. The store claims a 1Gi PersistentVolumeClaim, and it accepts an access key and secret key that the chart sets to published defaults. + +> [!IMPORTANT] +> The bundled store is not durable. Its credentials are shared defaults that every installation receives. Snapshots hold agent conversation state, so [move a production installation to your own bucket](#move-snapshots-to-an-external-bucket) before it carries real traffic. + +### Move snapshots to an external bucket + +1. Turn the bundled store off and select the backend in your Agent Substrate Helm values. Setting `atelet.storageBackend` to `s3` selects the S3 client, and any other value selects Google Cloud Storage. + + ```yaml + rustfs: + enabled: false + atelet: + storageBackend: s3 + ``` + +2. Supply the credentials that your backend needs. Setting `rustfs.enabled` to `false` stops the chart from setting the `AWS_*` variables, so neither component receives an endpoint or a key from Helm. Each storage client reads its credentials from the pod's own environment instead. + + - For Google Cloud Storage, the client uses application default credentials. Bind the components' ServiceAccounts through Workload Identity and set nothing further. + - For Amazon S3, the client reads the standard AWS environment and uses a role that the pod assumes. No static key is needed. + - For an S3-compatible store, pass `AWS_ENDPOINT_URL`, `AWS_S3_USE_PATH_STYLE`, and the static credentials through `atelet.extraEnv`. + + ```yaml + atelet: + extraEnv: + - name: AWS_ENDPOINT_URL + value: https://objects.example.com + - name: AWS_S3_USE_PATH_STYLE + value: "true" + ``` -The first place is the storage backend on the Agent Substrate components. Both the `atelet` DaemonSet and the `ate-api-server` Deployment read the same environment variables, and both must agree, because one writes snapshots and the other manages them. + -The following variables select the backend. `ATE_STORAGE_BACKEND` chooses the client, and the rest configure it. Leaving `ATE_STORAGE_BACKEND` unset selects Google Cloud Storage through application default credentials rather than disabling snapshots. +3. Point each Harness at the new bucket. + + ```yaml + spec: + substrate: + workerPoolRef: + name: kagent-default + snapshotPolicy: + location: s3://ate-snapshots/kagent/ + ``` + +4. Run a turn on an agent, suspend its Actor, and compare the `latestSnapshot` path [in the response](#inspect-the-runtime) against the `location` that you set. Because the backend and the location are configured separately, a mismatch between them breaks resume rather than install. Confirm the result rather than the intent. + +### How the chart reaches the components + +The `atelet` DaemonSet writes a snapshot, and the `ate-api-server` Deployment manages the snapshot. The chart renders `atelet.storageBackend` as the `ATE_STORAGE_BACKEND` variable on both workloads, so one Helm value keeps both in agreement. + +Reading these variables off a running pod helps when a snapshot fails to write. Edit the Helm values rather than the pod environment because the next Helm upgrade overwrites a direct edit. | Variable | Purpose | | -------- | ------- | @@ -151,18 +206,9 @@ The following variables select the backend. `ATE_STORAGE_BACKEND` chooses the cl | `AWS_S3_USE_PATH_STYLE` | `true` selects path-style addressing, which most S3-compatible stores need. | | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` | Static credentials for the S3 client. | -The second place is the location on each {{< gloss "Harness" >}}Harness{{< /gloss >}}, which names the bucket and prefix that its Actors' snapshots are written under. The field is required, and it must point into a bucket that the configured backend can reach. +### Snapshot layout in the bucket -```yaml -spec: - substrate: - workerPoolRef: - name: kagent-default - snapshotPolicy: - location: s3://ate-snapshots/kagent/ -``` - -Agent Substrate builds a path per Actor under that prefix, so one bucket serves many Harnesses without collision. The `` segment is the Actor's `metadata.uid`, not the `actorId` that `GetSubstrateStatus` reports. Both values are UUIDs, so read `latestSnapshot` off the Actor rather than matching a path to an Actor by eye. +Agent Substrate builds a path per Actor under the Harness prefix, so one bucket serves many Harnesses without collision. The `` segment is the Actor's `metadata.uid`, not the `actorId` that `GetSubstrateStatus` reports. Both values are UUIDs, so read `latestSnapshot` off the Actor rather than matching a path to an Actor by eye. ```console s3://ate-snapshots/kagent/atespaces//actors//snapshots/ @@ -176,49 +222,72 @@ s3://ate-snapshots/kagent/atespaces//tags/ A backup or lifecycle policy must cover both prefixes, because one scoped to `actors/` omits every checkpoint and understates how much the bucket holds. -Because the two places are configured independently, confirm the result rather than the intent. The `latestSnapshot` path in the [Inspect the runtime](#inspect-the-runtime) response is the bucket that an Actor last wrote to, so compare it against the `location` that you set. +## Keep pools on the sandbox class they serve -> [!WARNING] -> A development installation points at an in-cluster object store with well-known credentials, and it is not durable. Snapshots hold agent conversation state, so a production installation needs a real bucket, credentials that are not shared defaults, and a backup policy that matches how much conversation history you are willing to lose. +kagent configures the sandbox class for you. A default installation runs the `gvisor` class, and every {{< gloss "ActorTemplate" >}}ActorTemplate{{< /gloss >}} that kagent generates inherits the class of the pool that hosts it. Edit this setting only to move a pool to the `microvm` class or to change a pool's Worker build. For what each class isolates, see [Sandboxing]({{< link path="about/substrate-runtime/sandboxing#sandbox-classes" >}}). -## Keep pools on the sandbox class they serve +One rule governs the change. A pool's class, its Worker image, and the cluster's SandboxConfig must all name the same sandbox class, and Agent Substrate rejects the Actor rather than falling back when they disagree. + +### Set a pool's sandbox class -A pool's sandbox class decides which sandbox runtime its Workers provide, and kagent constrains the choice more tightly than Agent Substrate does. +1. Set the class and the matching Worker build in your kagent Helm values. -Agent Substrate supports the `gvisor` and `microvm` classes, as explained in [Sandboxing]({{< link path="about/substrate-runtime/sandboxing" >}}). kagent compiles every ActorTemplate to the class that its pool selected and to the matching SandboxConfig: a pool on `gvisor` (or no `sandboxClass`, which defaults to `gvisor`) gets the `gvisor` class and a SandboxConfig named `gvisor-default`, while a pool on `microvm` gets the `microvm` class and a SandboxConfig named `microvm`. Placement never relaxes the class constraint: a Worker in a pool whose class does not match an ActorTemplate's class reports `RevisionInvalid` with `unsupported sandbox class ""`, and placement of this Actor fails with `WorkerPoolNotFound`. + {{< tabs >}} + {{% tab name="gvisor" %}} + ```yaml + substrateWorkerPool: + sandboxClass: gvisor + workerImage: "ghcr.io/kagent-dev/substrate/ateom-gvisor:v{{< reuse "kagent-docs/versions/agent-substrate.md" >}}" + ``` + {{% /tab %}} + {{% tab name="microvm" %}} + ```yaml + substrateWorkerPool: + sandboxClass: microvm + workerImage: "ghcr.io/kagent-dev/substrate/ateom-microvm:v{{< reuse "kagent-docs/versions/agent-substrate.md" >}}" + ``` + {{% /tab %}} + {{< /tabs >}} -Leave a pool that backs kagent Harnesses on the class you intend to run, and keep the pool's image on the matching Worker build: `ateom-gvisor` for a pool on `gvisor`, `ateom-microvm` for a pool on `microvm`. + | Field | Description | + | ----- | ----------- | + | `sandboxClass` | The sandbox runtime family for this pool's Workers, and for every ActorTemplate that kagent generates for the pool. Omitting the field selects `gvisor`. | + | `workerImage` | The Worker build that provides the runtime. `ateom-gvisor` serves the `gvisor` class and `ateom-microvm` serves `microvm`. A build that does not match its pool's class never registers a Worker. | -```yaml -substrateWorkerPool: - sandboxClass: gvisor - workerImage: "ghcr.io/kagent-dev/substrate/ateom-gvisor:v{{< reuse "kagent-docs/versions/agent-substrate.md" >}}" -``` +2. Confirm that the cluster has a SandboxConfig for this class. kagent names the configuration directly rather than resolving a default, so a `gvisor` pool needs a config named `gvisor-default` and a `microvm` pool needs a config named `microvm`. -```yaml -substrateWorkerPool: - sandboxClass: microvm - workerImage: "ghcr.io/kagent-dev/substrate/ateom-microvm:v{{< reuse "kagent-docs/versions/agent-substrate.md" >}}" -``` + ```bash + kubectl get sandboxconfigs + ``` -> [!NOTE] -> The `ateomImage` field in the [Inspect the runtime](#inspect-the-runtime) response reports this same setting, which the WorkerPool resource calls `workerImage`. To check which build a pool is running, compare the two names. + Example output: + ```console + NAME CLASS AGE + gvisor-default gvisor 26h + microvm microvm 26h + ``` -A cluster-scoped SandboxConfig for the pool's class must also exist, because kagent names it directly rather than resolving a default: `gvisor-default` for the `gvisor` class and `microvm` for the `microvm` class. A missing one fails template preparation with `SandboxConfig "" not found`. On a default installation only the `gvisor` class is wired up: Substrate applies `sandboxconfig-gvisor.yaml`, and kagent installs nothing for `microvm`. A `microvm` pool therefore needs operator-staged assets: the `microvm` SandboxConfig is a template that the Substrate installer stages out of band (for example via `hack/install-microvm-deps.sh --install`), not something kagent provides. + A default installation creates only the `gvisor` class, because the Agent Substrate chart applies `sandboxconfig-gvisor.yaml` and kagent installs nothing for `microvm`. The `microvm` configuration and its runtime assets are staged out of band by the Agent Substrate installer, such as through `hack/install-microvm-deps.sh --install`. -```bash -kubectl get sandboxconfigs -``` -Example output: -```console -NAME CLASS AGE -gvisor-default gvisor 26h -microvm microvm 26h -``` +3. Verify that the pool's Workers came up on the build that you set. The `ateomImage` field in the [runtime response](#inspect-the-runtime) reports the image that each Worker is running, which the WorkerPool resource calls `workerImage`. Compare the two, and confirm in the ouput that the pool reports its full replica count as ready. + + ```bash + kubectl get workerpools -n kagent + ``` + +### Diagnose a class mismatch + +Each of the three settings fails differently when it disagrees with the others. + +- A Worker whose pool class does not match an ActorTemplate's class reports `RevisionInvalid` with `unsupported sandbox class ""`, and placement of this Actor fails with `WorkerPoolNotFound`. +- A missing SandboxConfig fails template preparation with `SandboxConfig "" not found`. +- A Worker image built for another class never registers, so the pool reports fewer ready Workers than its replica count. + +### Considerations before changing a running pool -Two further constraints apply when you change any of this on a running cluster. +Two constraints apply to a pool that already serves traffic. - **Snapshots do not move between sandbox classes.** An Actor cannot resume from a snapshot that a different class produced, so changing the class of a pool that holds live Actors strands their state. -- **Nodes fetch sandbox runtime assets from a public Google Cloud Storage URL**, named in the SandboxConfig, independently of the storage backend that you configured for snapshots. A cluster with restricted egress needs a path to that URL, or a SandboxConfig that points at a copy you host. +- **Nodes fetch sandbox runtime assets from a public Google Cloud Storage URL**, named in the SandboxConfig, independently of the storage backend that you configured for snapshots. A cluster with restricted egress needs a path to this URL, or a SandboxConfig that points at a copy you host. Replacing `workerImage` rolls the Worker pods through the generated Deployment. The pods take a one-hour termination grace period, so an in-flight turn finishes rather than being cut off, and a rollout of a busy pool takes as long as its longest running turn. From c8e9d031fd4284fcb4224ab28578480a1b9242a8 Mon Sep 17 00:00:00 2001 From: Rachael Graham Date: Mon, 12 Oct 2026 14:48:57 -0500 Subject: [PATCH 07/10] Clarify kagent 1.0 core concepts Refine the explanations of migration, Harness and Agent resources, and Session lifecycle. Add an overview of Actor compute, identity, and isolation, with links to the detailed runtime docs. Signed-off-by: Rachael Graham --- .../content/kagent/1.x/about/core-concepts.md | 51 +++++++++++++------ 1 file changed, 36 insertions(+), 15 deletions(-) diff --git a/docs-site/content/kagent/1.x/about/core-concepts.md b/docs-site/content/kagent/1.x/about/core-concepts.md index cda529b7..3e0f7227 100644 --- a/docs-site/content/kagent/1.x/about/core-concepts.md +++ b/docs-site/content/kagent/1.x/about/core-concepts.md @@ -7,7 +7,9 @@ author: kagent.dev ## kagent 1.0 -{{< reuse "kagent-docs/snippets/name-product.md" >}} 1.0 replaces 0.x's Deployment-based `Agent` custom resource with a new model built around **Harness**, **AgentTemplate**, **Agent**, and **Session**, running on [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}) instead of the plain Kubernetes Deployments that the 0.x model uses. The 1.0 `Agent` kind shares the 0.x name but not its meaning: it pairs an AgentTemplate with a Harness rather than describing a Deployment. This page defines the vocabulary that the rest of the 1.0 model docs use. If you already have a 0.x installation, you migrate to the new model rather than upgrading into it. No controller converts a 0.x resource, and the two API groups share no conversion, so you author the 1.0 resources yourself and retire the 0.x ones. [Migrate from 0.x]({{< link path="operations/upgrade-from-0x#recreate-your-resources" >}}) maps each 0.x resource onto its 1.0 replacement. +{{< reuse "kagent-docs/snippets/name-product.md" >}} 1.0 replaces 0.x's Deployment-based `Agent` custom resource with a new model built around **Harness**, **AgentTemplate**, **Agent**, and **Session**. The 1.0 `Agent` kind keeps the 0.x name but not its meaning, because it pairs an AgentTemplate with a Harness rather than describing a Deployment. This page defines the vocabulary that the rest of the 1.0 documentation uses. + +Moving a 0.x installation to 1.0 is a manual migration, and no upgrade path exists. No controller converts a 0.x resource, and the two API groups share no conversion, so you author the 1.0 resources yourself and retire the 0.x ones. [Migrate from 0.x]({{< link path="operations/upgrade-from-0x#recreate-your-resources" >}}) maps each 0.x resource onto its 1.0 replacement. The new model separates what an agent can do from how it is allowed to run, and then names the pairing explicitly: @@ -17,7 +19,11 @@ The new model separates what an agent can do from how it is allowed to run, and - A [**Session**](#session) is a running conversation with an Agent. - An [**Actor**](#actor) is the sandboxed process, provided by Substrate, that a Session runs on. -The following diagram shows how an Agent becomes a running conversation. The kagent controller compiles each Agent into an {{< gloss "ActorTemplate" >}}ActorTemplate{{< /gloss >}}, and each Session is created from that ActorTemplate and runs on an Actor. +Agents do not run as Kubernetes Deployments. kagent runs every conversation on [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}), a compute layer that starts capacity in advance and places sandboxed workloads onto it. Substrate keeps a conversation alive without keeping it resident. When a conversation goes idle, Substrate saves its state to a {{< gloss "Snapshot" >}}snapshot{{< /gloss >}} and releases the capacity it held, then restores it from that snapshot when the next message arrives. A caller never sees the pause. + +Substrate needs its own definition of the workload to run, so the kagent controller automatically compiles each Agent into an **{{< gloss "ActorTemplate" >}}ActorTemplate{{< /gloss >}}**, the Substrate resource that an Actor is created from. + +The following diagram shows how an Agent becomes a running conversation.

```mermaid @@ -48,16 +54,18 @@ An operator applies the three custom resources directly. The kagent controller w ## Harness + + An agent harness is the application layer that executes an agent and adds capabilities around its model. It typically assembles context, invokes the model, executes tool calls, loads skills, feeds results back into the agent loop, and manages session state. A harness can also provide memory, subagents, permissions, approvals, sandboxing, and event streaming. Examples include [Claude Code](https://code.claude.com/docs/en/how-claude-code-works), [Codex CLI](https://learn.chatgpt.com/docs/codex/cli), [Gemini CLI](https://geminicli.com/docs/), [OpenClaw](https://docs.openclaw.ai/concepts/agent-runtimes), and custom in-house applications. Each of those runs on a local machine with your own shell and files. A **Harness** in kagent is a Kubernetes custom resource that applies that idea to a cluster. It defines _how an agent is allowed to run_, and specifies: -- **Runtime**: The engine that executes the agent. A Harness selects exactly one of `kagent`, `codex`, `claude`, or `byo`, and kagent compiles all four. `kagent` runs kagent's own Go and Python engines, `codex` and `claude` run those coding agents, and `byo` runs any image that implements kagent's A2A contract. +- **Runtime**: The engine that executes the agent. A Harness selects exactly one of `kagent`, `codex`, `claude`, or `byo`, and kagent compiles all four. `kagent` runs kagent's own agent engine, `codex` and `claude` run those coding agents, and `byo` runs any image that implements kagent's A2A contract. - **Workload**: The container image, command, and arguments that the runtime runs as. - **Environment**: Literal environment values for the runtime, set in `spec.env`. -- **Substrate policy**: The capacity that the agent runs on, and where its state is kept when it goes idle. Agents do not run as Deployments. [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}), which is the compute layer underneath kagent, runs each conversation as a sandboxed process on pre-started capacity called a [WorkerPool]({{< link path="about/architecture/agent-substrate#workers-and-workerpools" >}}). A {{< gloss "Snapshot" >}}snapshot{{< /gloss >}} is the saved state of one of those sandboxed processes. A snapshot is written when the conversation goes idle so that its capacity returns to the pool, and read back when the next message arrives. The Harness names the WorkerPool to schedule onto and where those snapshots are stored. +- **Substrate policy**: The capacity that the agent runs on, and where its state is kept when it goes idle. The Harness names a [WorkerPool]({{< link path="about/architecture/agent-substrate#workers-and-workerpools" >}}) to schedule onto and the storage location that snapshots write to. A Harness names no AgentTemplate, and an AgentTemplate names no Harness. An [Agent](#agent) pairs the two, and nothing pairs them implicitly. @@ -85,16 +93,9 @@ For the complete AgentTemplate schema, see the [API reference]({{< link path="re ## Agent -An **Agent** is a Kubernetes custom resource that pairs _one AgentTemplate with one Harness_. The pairing is explicit: each side takes either a reference to an existing resource or a complete inline spec, and exactly one of the two per side. +An **Agent** is a Kubernetes custom resource that pairs _one AgentTemplate with one Harness_. Each side of the pairing either names an existing resource or carries a complete spec written inline, and an Agent sets exactly one of the two per side. -| Field | What it selects | -| ----- | --------------- | -| `spec.templateRef` | An existing AgentTemplate in the Agent's namespace, by name | -| `spec.template` | A complete AgentTemplate spec, written inline | -| `spec.harnessRef` | An existing Harness in the Agent's namespace, by name | -| `spec.harness` | A complete Harness spec, written inline | - -The template side and the harness side are set independently, so an Agent can reference both, inline both, or mix the two. An inline spec is a complete value rather than an override of a referenced one, and kagent creates no Kubernetes object to back it. Every reference, including one that is nested inside an inline spec, resolves in the Agent's own namespace. +The two sides are set independently. An Agent can name both resources, inline both, or mix the two. An inline spec is a complete definition, not an override of a named resource. kagent creates no Kubernetes object to back one. Every name resolves in the Agent's own namespace, including a name nested inside an inline spec. For the fields that express each choice, see the [API reference]({{< link path="reference/api-ref#agent" >}}). The following Agent resource references both sides: @@ -135,13 +136,33 @@ kagent agent session create --agent assistant -n kagent After it is created, a Session talks to callers over the {{< gloss "A2A" >}}A2A{{< /gloss >}} (Agent-to-Agent) protocol, through kagent's A2A gateway. Callers address the Agent rather than the Session. The HTTP endpoint is `/agents/{namespace}/{name}` and gRPC carries the same `namespace/name` in the standard A2A `tenant` field. The Session's ID is the A2A `contextId`, so a message that carries no context identifier starts a new conversation, and a message that repeats one continues that conversation. -A Session that records no task activity for seven days is deleted by an expiration worker. The `controller.sessionIdleTTL` Helm value sets that window. A value of `0` turns the expiration worker off. For more information about what the deletion retains, see [Expire idle conversations]({{< link path="operations/operational-considerations#expire-idle-sessions" >}}). +A Session that records no task activity for seven days is deleted by an expiration worker. The `controller.sessionIdleTTL` Helm value sets that window. A value of `0` turns the expiration worker off. For more information about what the deletion retains, see [Expire idle Sessions]({{< link path="operations/operational-considerations#expire-idle-sessions" >}}). ## Actor An **Actor** is the sandboxed unit of compute, provided by [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}), that _runs a Session's conversation loop_. Every Session is backed by an Actor. -Actors are the reason why Sessions can suspend and resume cheaply instead of staying resident. An idle Actor can be snapshotted and torn down, then resumed from that snapshot on demand. To understand the full mechanics ({{< gloss "Worker" >}}Workers{{< /gloss >}}, {{< gloss "WorkerPool" >}}WorkerPools{{< /gloss >}}, ActorTemplates, and snapshotting), see [Agent Substrate architecture]({{< link path="about/architecture/agent-substrate" >}}). +An Actor lets a Session suspend and resume cheaply instead of staying resident. To understand the {{< gloss "Worker" >}}Workers{{< /gloss >}} and {{< gloss "WorkerPool" >}}WorkerPools{{< /gloss >}} that Actors run on, see [Agent Substrate architecture]({{< link path="about/architecture/agent-substrate" >}}). + +## Compute, identity, and isolation + +The Harness, AgentTemplate, and Agent resources describe what an agent is. The Actor behind each Session decides how that agent runs. An Actor supplies the agent's compute, its network identity, and the sandbox that confines it. + +### Compute + +An Actor occupies a slot on a Worker, which is a pre-started pod in a WorkerPool. A Worker hosts many Actors at once, and an Actor that goes idle is suspended to a snapshot so that its slot returns to the pool. A pool therefore serves far more Sessions than it holds Workers. Agent Substrate also pauses an Actor to a node-local snapshot, resumes it onto whichever Worker has room, and forks a tagged snapshot into a second Actor that starts from the same state. For these operations and what each snapshot keeps, see [Suspend and resume]({{< link path="about/substrate-runtime/suspend-and-resume" >}}). + +### Identity + +Each Actor carries its own cryptographic identity that is issued as a certificate and scoped to the {{< gloss "Atespace" >}}atespace{{< /gloss >}} that it belongs to. Agent Substrate's egress gateway authorizes every outbound connection against the identity on the certificate rather than against anything in the request. Because of this, an agent cannot reach a destination by claiming to be another agent. + +This identity is also what keeps API keys out of the agent. An agent sends a placeholder header, and the gateway replaces it with the real credential as the request passes through, so the key never enters the agent's environment or its filesystem. For the allowlist that the gateway enforces and how credentials are attached to it, see [Egress control]({{< link path="about/substrate-runtime/networking-and-egress" >}}). + +### Isolation + +Each Actor runs in its own sandbox, with its own filesystem and its own durable volume for application data. Two Sessions on the same Agent share no files and cannot see any of each other's state, even when they run on the same Worker. One conversation's data therefore stays out of another's context. The sandbox also isolates the workload's system calls from the node's kernel, which matters because an agent runs commands that a model chose. For what each sandbox class separates, see [Sandboxing]({{< link path="about/substrate-runtime/sandboxing" >}}). + + ## Subagent tools From f315aaa22a6d78a3c3596f76e6ff9d1d12d36f2a Mon Sep 17 00:00:00 2001 From: Rachael Graham Date: Mon, 12 Oct 2026 15:14:04 -0500 Subject: [PATCH 08/10] fixes Signed-off-by: Rachael Graham --- .../content/kagent/1.x/about/core-concepts.md | 5 +- .../operations/operational-considerations.md | 74 +++++++++++-------- 2 files changed, 46 insertions(+), 33 deletions(-) diff --git a/docs-site/content/kagent/1.x/about/core-concepts.md b/docs-site/content/kagent/1.x/about/core-concepts.md index 3e0f7227..5d94f787 100644 --- a/docs-site/content/kagent/1.x/about/core-concepts.md +++ b/docs-site/content/kagent/1.x/about/core-concepts.md @@ -150,7 +150,10 @@ The Harness, AgentTemplate, and Agent resources describe what an agent is. The A ### Compute -An Actor occupies a slot on a Worker, which is a pre-started pod in a WorkerPool. A Worker hosts many Actors at once, and an Actor that goes idle is suspended to a snapshot so that its slot returns to the pool. A pool therefore serves far more Sessions than it holds Workers. Agent Substrate also pauses an Actor to a node-local snapshot, resumes it onto whichever Worker has room, and forks a tagged snapshot into a second Actor that starts from the same state. For these operations and what each snapshot keeps, see [Suspend and resume]({{< link path="about/substrate-runtime/suspend-and-resume" >}}). +An Actor occupies a slot on a Worker, which is a pre-started pod in a WorkerPool. A Worker hosts many Actors at once, and an Actor that goes idle is suspended to a snapshot so that its slot returns to the pool. A pool therefore serves far more Sessions than it holds Workers. Agent Substrate also pauses an Actor to a node-local snapshot, and resumes an Actor onto whichever Worker has room. For these operations and what each snapshot keeps, see [Suspend and resume]({{< link path="about/substrate-runtime/suspend-and-resume" >}}). + +> [!NOTE] +> Forking a {{< gloss "Checkpoint" >}}checkpoint{{< /gloss >}} into a second Session is unavailable in this release. kagent captures process memory in every snapshot, and a restored copy would hold the source Session's identifiers, so `ForkSession` rejects the request. Forking returns once Agent Substrate separates the pause and suspend snapshot scopes again. ### Identity diff --git a/docs-site/content/kagent/1.x/operations/operational-considerations.md b/docs-site/content/kagent/1.x/operations/operational-considerations.md index f50f43de..d9f557b7 100644 --- a/docs-site/content/kagent/1.x/operations/operational-considerations.md +++ b/docs-site/content/kagent/1.x/operations/operational-considerations.md @@ -5,67 +5,75 @@ weight: 10 author: kagent.dev --- -A default {{< reuse "kagent-docs/snippets/name-product.md" >}} installation runs one controller replica against a bundled database, and schedules agents onto any node in the cluster. Each of those defaults works for a trial installation, but all three need a decision before you run {{< reuse "kagent-docs/snippets/name-product.md" >}} in production. This page covers which database to run, how many controller replicas to keep, and how to treat the nodes that host Workers. +A default {{< reuse "kagent-docs/snippets/name-product.md" >}} installation runs one controller replica against the development database that `kagent install` creates, and schedules agents onto any node in the cluster. Each of those defaults works for a trial installation, but all three need a decision before you run {{< reuse "kagent-docs/snippets/name-product.md" >}} in production. This page covers which database to run, how many controller replicas to keep, and how to treat the nodes that host Workers. ## Choose a database -kagent stores conversations, {{< gloss "Session" >}}Sessions{{< /gloss >}}, and compiled {{< gloss "Revision" >}}revisions{{< /gloss >}} in PostgreSQL. A bundled instance ships with the chart and runs by default, so an installation needs no external database to start. Run your own PostgreSQL instance for production, because the bundled one is a single pod backed by one PersistentVolumeClaim (PVC). +kagent stores conversations, {{< gloss "Session" >}}Sessions{{< /gloss >}}, and compiled {{< gloss "Revision" >}}revisions{{< /gloss >}} in PostgreSQL. The kagent chart deploys no database of its own. It reads a connection string from a Kubernetes Secret, and `database.postgres.connectionStringSecretRef` names that Secret and the key inside it. -Two independent settings determine what runs and what the controller talks to: +```yaml +database: + postgres: + connectionStringSecretRef: + name: kagent-postgres + key: connectionString +``` -- `database.postgres.bundled.enabled` determines whether the chart deploys the bundled PostgreSQL pod and its PersistentVolumeClaim (PVC). This setting does not affect which database the controller connects to. -- `database.postgres.url` and `database.postgres.urlFile` determine what the controller connects to. When either is set, the controller uses it. When both are empty, the controller connects to the bundled instance. +Run your own PostgreSQL instance for production, so that the data outlives the cluster that kagent runs on. -The controller resolves its connection in the order `urlFile`, then `url`, then the bundled connection string. Because the two settings are independent, a bundled pod can keep running while the controller points at an external database, which gives you a window to migrate data across. +### The development database -| `bundled.enabled` | `url` or `urlFile` | Bundled pod deployed | Controller connects to | -| ----------------- | ------------------ | -------------------- | ---------------------- | -| `true` | Omitted | Yes | The bundled instance | -| `false` | Set | No | The external instance | -| `true` | Set | Yes | The external instance | -| `false` | Omitted | No | Nothing. The chart fails to render. | +`kagent install` creates a database for you unless you pass `--skip-database-setup`. It deploys a single-replica `kagent-postgresql` Deployment backed by a 500Mi PersistentVolumeClaim, creates the roles that kagent and Agent Substrate connect as, and writes the connection Secrets for both: `kagent-postgres` in the kagent namespace, and `substrate-postgres-readwrite` and `substrate-postgres-owner` in `ate-system`. -The last row fails at template time with `No database connection configured`, rather than installing a controller that cannot start. +Connections to this database carry no password. Each client authenticates with a Pod Certificate projected into its pod, and the connection string points `sslcert` and `sslkey` at the files under `/run/postgres.podcert.ate.dev`. `database.postgres.clientCertificate.enabled` projects those files, and `kagent install` sets it. -The bundled instance claims its PersistentVolumeClaim from the cluster's default StorageClass. Set `database.postgres.bundled.storageClassName` to choose a different one. +> [!IMPORTANT] +> The development database is a single pod backed by one PersistentVolumeClaim, and `kagent install` sizes that claim at 500Mi. Losing the pod's volume loses every Session and every compiled revision, so move a production installation to a PostgreSQL instance of your own. ### Use an external PostgreSQL instance For production, run PostgreSQL outside the cluster's lifecycle so that a kagent uninstall cannot delete the data. -1. Set the connection in your Helm values file. To keep credentials out of Helm values, mount the connection string from a Kubernetes Secret and reference the mount path by using `urlFile`. +1. Create a Secret that holds the connection string. + + ```bash + kubectl create secret generic my-postgres \ + --namespace kagent \ + --from-literal=connectionString="postgresql://user:password@postgres.example.com:5432/kagent?sslmode=verify-full" + ``` + +2. Point the chart at your Secret in your Helm values file. ```yaml database: postgres: - urlFile: /var/secrets/db-url + connectionStringSecretRef: + name: my-postgres + key: connectionString + clientCertificate: + enabled: false + role: kagent_owner + schema: kagent vectorEnabled: true skipMigrations: false pool: maxConns: 20 minConns: 2 - bundled: - enabled: false controller: replicas: 3 - volumes: - - name: db-secret - secret: - secretName: my-postgres-url-secret - volumeMounts: - - name: db-secret - mountPath: /var/secrets - readOnly: true ``` | Setting | Description | | ------- | ----------- | - | `database.postgres.url` or `database.postgres.urlFile` | The connection string, or the path to a file holding it. `urlFile` keeps the credentials out of Helm values. | - | `database.postgres.vectorEnabled` | Set to `true` only when the instance has the `pgvector` extension installed. The setting enables the vector migration, and features that depend on it, such as [long-term memory]({{< link path="agents/agent-memory" >}}), fail without the extension. The bundled image does not include `pgvector`. | + | `database.postgres.connectionStringSecretRef` | The Secret and key that hold the connection string. Keeping the string in a Secret keeps the credentials out of Helm values. The controller restarts to pick up a changed Secret. | + | `database.postgres.clientCertificate` | Set `enabled` to `true` only for a database that authenticates clients with Pod Certificates, such as the one that `kagent install` creates. The connection string must then reference the files under `/run/postgres.podcert.ate.dev`. | + | `database.postgres.role` | The role that kagent assumes on each connection. Give each installation that shares one database a distinct role. | + | `database.postgres.schema` | The schema that holds kagent's tables. `database.postgres.vectorSchema` names the schema that holds the shared `pgvector` extension, and kagent needs `USAGE` on it. | + | `database.postgres.vectorEnabled` | Set to `true` only when the instance has the `pgvector` extension installed. The setting enables the vector migration, and features that depend on it, such as [long-term memory]({{< link path="agents/agent-memory" >}}), fail without the extension. The development database does not include `pgvector`. | | `database.postgres.pool` | Connection pool sizing, through `maxConns`, `minConns`, `maxConnIdleTime`, and `maxConnLifetime`. Omit the fields to keep the pgx library defaults. | | `database.postgres.skipMigrations` | Set to `true` to stop the controller from running migrations at startup. The controller then verifies that the database is already migrated and fails if it is not. Apply the migrations from a pipeline before you install or upgrade. | -2. Apply the values to the kagent Helm release. +3. Apply the values to the kagent Helm release. ```bash helm upgrade --install kagent \ @@ -87,7 +95,7 @@ controller: Leader election keeps the replicas from conflicting. One replica holds a Kubernetes lease and performs reconciliation, garbage collection, and scheduled runs. The other replicas stay ready and take over when the leader's lease expires. > [!IMPORTANT] -> If an installation runs on the bundled database, extra controller replicas do not make the installation highly available. The bundled PostgreSQL instance is a single pod backed by one PVC, so losing it stops every replica at once. Run an external PostgreSQL instance before counting controller replicas toward availability. +> If an installation runs on the development database, extra controller replicas do not make the installation highly available. That database is a single pod backed by one PersistentVolumeClaim, so losing it stops every replica at once. Run an external PostgreSQL instance before counting controller replicas toward availability. ## Node operations @@ -142,12 +150,14 @@ controller: No per-agent override exists, so this one value governs every Session in the installation. No maximum exists either, so a longer value is accepted. -Idle time runs from whichever is later: the Session's creation, or its most recently stored A2A event. Reads, renames, lifecycle calls, and retried writes do not reset it, so a Session that is only ever listed still expires. A {{< gloss "Fork" >}}fork{{< /gloss >}} keeps its source's event timestamps. Its own creation time gives it a full lifetime, so it does not inherit an almost-expired one. +Idle time runs from whichever is later: the Session's creation, or its most recently stored A2A event. Reads, renames, lifecycle calls, and retried writes do not reset it, so a Session that is only ever listed still expires. + + Work in progress is never deleted. A Session with a running task survives past the interval, and so does one with a turn waiting at `INPUT_REQUIRED` or `AUTH_REQUIRED`. A pending lifecycle operation or a checkpoint being captured also holds a Session open. > [!IMPORTANT] -> **Expiry bounds Sessions, not history.** Deleting an idle Session removes the Session, its shares, its runtime row, and its creation receipt, so `GetSession` returns not-found afterward. The A2A context, its tasks, its event history, and any explicit {{< gloss "Checkpoint" >}}checkpoints{{< /gloss >}} stay in PostgreSQL, and a retained checkpoint can still be forked after the Session it was taken on has expired. kagent has no setting that bounds this retained history, so size a database for a long-running installation with the audit history planned separately. +> **Expiry bounds Sessions, not history.** Deleting an idle Session removes the Session, its shares, its runtime row, and its creation receipt, so `GetSession` returns not-found afterward. The A2A context, its tasks, its event history, and any explicit {{< gloss "Checkpoint" >}}checkpoints{{< /gloss >}} stay in PostgreSQL, and a retained checkpoint outlives the Session it was taken on. kagent has no setting that bounds this retained history, so size a database for a long-running installation with the audit history planned separately. An expired Session releases its creation request ID, unlike an explicit client deletion, which keeps a tombstone. The same caller can reuse that request ID to start a fresh Session. From 2d35aa4797993ebb70aa3f572f6571571613d149 Mon Sep 17 00:00:00 2001 From: Rachael Graham Date: Tue, 13 Oct 2026 09:53:33 -0500 Subject: [PATCH 09/10] review Signed-off-by: Rachael Graham --- .../1.x/about/architecture/agent-substrate.md | 4 +++- .../content/kagent/1.x/about/core-concepts.md | 3 ++- .../substrate-runtime/networking-and-egress.md | 10 +++++----- .../substrate-runtime/suspend-and-resume.md | 4 ++-- .../1.x/operations/operational-considerations.md | 16 +++++++--------- .../1.x/operations/tune-agent-substrate.md | 16 ++++++++-------- 6 files changed, 27 insertions(+), 26 deletions(-) diff --git a/docs-site/content/kagent/1.x/about/architecture/agent-substrate.md b/docs-site/content/kagent/1.x/about/architecture/agent-substrate.md index 63b86451..b98985ea 100644 --- a/docs-site/content/kagent/1.x/about/architecture/agent-substrate.md +++ b/docs-site/content/kagent/1.x/about/architecture/agent-substrate.md @@ -38,7 +38,9 @@ One Worker hosts many Actors at the same time. When each Worker starts, it repor Substrate checks two things before it places an Actor on a Worker: * Whether the Worker has a free Actor slot. This check always runs. -* Whether the Worker has enough spare CPU and memory. This check runs only when the ActorTemplate declares what one Actor needs, and kagent declares nothing for a Session. The free Actor slot check is therefore the only one that decides where a Session's Actor lands. [Worker resource limits]({{< link path="operations/tune-agent-substrate#set-resource-requests-on-workers" >}}) still matter, because a Worker with no limits reports its whole node as its own capacity. +* Whether the Worker has enough spare CPU and memory. This check runs only when the ActorTemplate states how much CPU and memory one Actor needs. + +An ActorTemplate that the kagent controller compiles states no per-Actor CPU or memory, so the free Actor slot check is the only one that decides where a Session's Actor lands. [Worker resource limits]({{< link path="operations/tune-agent-substrate#set-resource-requests-on-workers" >}}) still matter, because a Worker with no limits reports its whole node as its own capacity. diff --git a/docs-site/content/kagent/1.x/about/core-concepts.md b/docs-site/content/kagent/1.x/about/core-concepts.md index 5d94f787..a6ebeb97 100644 --- a/docs-site/content/kagent/1.x/about/core-concepts.md +++ b/docs-site/content/kagent/1.x/about/core-concepts.md @@ -9,7 +9,8 @@ author: kagent.dev {{< reuse "kagent-docs/snippets/name-product.md" >}} 1.0 replaces 0.x's Deployment-based `Agent` custom resource with a new model built around **Harness**, **AgentTemplate**, **Agent**, and **Session**. The 1.0 `Agent` kind keeps the 0.x name but not its meaning, because it pairs an AgentTemplate with a Harness rather than describing a Deployment. This page defines the vocabulary that the rest of the 1.0 documentation uses. -Moving a 0.x installation to 1.0 is a manual migration, and no upgrade path exists. No controller converts a 0.x resource, and the two API groups share no conversion, so you author the 1.0 resources yourself and retire the 0.x ones. [Migrate from 0.x]({{< link path="operations/upgrade-from-0x#recreate-your-resources" >}}) maps each 0.x resource onto its 1.0 replacement. +> [!NOTE] +> Moving a 0.x installation to 1.0 is a manual migration, and no upgrade path exists. No controller converts a 0.x resource, and the two API groups share no conversion, so you author the 1.0 resources yourself and retire the 0.x ones. [Migrate from 0.x]({{< link path="operations/upgrade-from-0x#recreate-your-resources" >}}) maps each 0.x resource onto its 1.0 replacement. The new model separates what an agent can do from how it is allowed to run, and then names the pairing explicitly: diff --git a/docs-site/content/kagent/1.x/about/substrate-runtime/networking-and-egress.md b/docs-site/content/kagent/1.x/about/substrate-runtime/networking-and-egress.md index 9308bf2b..37bff87b 100644 --- a/docs-site/content/kagent/1.x/about/substrate-runtime/networking-and-egress.md +++ b/docs-site/content/kagent/1.x/about/substrate-runtime/networking-and-egress.md @@ -16,11 +16,11 @@ For the inbound half of the picture, and for the NetworkPolicy that protects Wor ## How the gateway authorizes a connection -Outbound traffic leaves a Worker through the egress gateway rather than directly out. This chokepoint gives Agent Substrate one place to decide what each agent can reach. The gateway takes three steps on every connection, and each step depends on the one before it, because the Actor's identity selects the policy, and the policy supplies the rules. +All outbound traffic leaves a Worker through the egress gateway, which gives Agent Substrate one place to decide what each agent can reach. The gateway takes three steps on every connection, and each step depends on the one before it, because the Actor's identity selects the policy, and the policy supplies the rules. 1. The gateway identifies the Actor. It reads the Actor's SPIFFE ID from the client certificate on the connection from the Actor, and it verifies this certificate rather than trusting anything in the request. -2. The gateway fetches the `EgressPolicy` for this Actor. Each Actor has at most one `EgressPolicy` named `default`, and Agent Substrate addresses the policy by the Actor rather than by this name. An Actor with no policy reaches nothing, because a policy is the only thing that allows a destination. +2. The gateway fetches the `EgressPolicy` for this Actor. Each Actor has at most one `EgressPolicy` named `default`, and Agent Substrate addresses the policy by the Actor rather than by this name. The gateway denies every request from an Actor that has no policy, because only a policy allows access to a destination. 3. The gateway picks which rule applies to the request. Rules are an unordered set rather than a list, and exactly one rule applies to a given destination name and port regardless of its protocol. When more than one rule matches, the gateway applies two criteria in order, moving to the second only when the first ties. @@ -29,16 +29,16 @@ Outbound traffic leaves a Worker through the egress gateway rather than directly A request that no rule matches is denied, and only the rule that applies contributes a credential. -The verified identity holds the other two steps together. The Actor's identity is established only by the certificate, so the gateway authorizes every request against the Actor that actually sent it regardless of the headers or hostnames that the request carries. The gateway refuses a connection that does not present a certificate rather than falling back to a default identity. Each Actor also gets its own connection pool, even when two of them dial the same address, so that every connection carries requests from the one Actor that opened it. +Policy enforcement depends on this verified identity. The Actor's identity is established only by the certificate, so the gateway authorizes every request against the Actor that actually sent it regardless of the headers or hostnames that the request carries. The gateway refuses a connection that does not present a certificate rather than falling back to a default identity. Each Actor also gets its own connection pool, even when two of them dial the same address, so that every connection carries requests from the one Actor that opened it. > [!NOTE] > Two rules that tie on the same name and port are rejected when the policy is admitted, so a policy never reaches the gateway with an ambiguous match. A policy holds at most 256 rules. ### Traffic types -A rule's type fixes the kind of connection it covers and the place that the gateway reads the destination hostname from. Three types cover the three cases. +Each rule defines the type of connection that is allowed to a destination and how the gateway determines the destination's hostname. The following connection types are allowed: -- **`http`**: Cleartext HTTP, meaning HTTP/1.1 and the h2c that gRPC uses. The gateway reads the `Host` header and matches it against the rule's hostnames. TLS to those same names is not allowed by this rule. +- **`http`**: Cleartext HTTP, meaning HTTP/1.1 and the h2c that gRPC uses. The gateway reads the hostname from the `Host` header and matches it against the rule's hostnames. TLS to those same names is not allowed by this rule. - **`https`**: HTTPS, which the gateway intercepts. It terminates the connection with a certificate minted for the destination name, reads the request inside, matches it against the rule's hostnames, and re-originates its own TLS connection onward. HTTP/1.1 and HTTP/2 work inside this connection, and HTTP/3 is not supported. - **`tls_passthrough`**: TLS forwarded to the named servers without decryption. The gateway cannot read the request, so it decides at the TLS ClientHello instead. diff --git a/docs-site/content/kagent/1.x/about/substrate-runtime/suspend-and-resume.md b/docs-site/content/kagent/1.x/about/substrate-runtime/suspend-and-resume.md index 837ef171..b33b9941 100644 --- a/docs-site/content/kagent/1.x/about/substrate-runtime/suspend-and-resume.md +++ b/docs-site/content/kagent/1.x/about/substrate-runtime/suspend-and-resume.md @@ -7,7 +7,7 @@ aliases: - /kagent/1.x/substrate-runtime/suspend-and-resume/ --- -An agent spends most of its life waiting. It waits on a person to reply, and it waits on a large language model (LLM) to answer. [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}) runs each agent inside an **Actor**, the unit that it suspends and resumes. Rather than hold compute open through that waiting, Agent Substrate writes the idle Actor to a {{< gloss "Snapshot" >}}snapshot{{< /gloss >}} and releases the capacity that the Actor held on its {{< gloss "Worker" >}}Worker{{< /gloss >}}. The next message restores it. The {{< gloss "Session" >}}Session{{< /gloss >}} stays ready the whole time, so reclaiming this capacity costs the user nothing. This page explains what a snapshot captures, when kagent suspends an Actor, and what happens when a suspended Actor is addressed again. +An agent spends most of its life waiting. It waits on a person to reply, and it waits on a large language model (LLM) to answer. [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}) runs each agent inside an **Actor**, the unit that Agent Substrate suspends and resumes. To free compute resources while an agent waits, Agent Substrate writes the idle Actor to a {{< gloss "Snapshot" >}}snapshot{{< /gloss >}} and releases the capacity that the Actor held on its {{< gloss "Worker" >}}Worker{{< /gloss >}}. The next message to the agent restores the Actor. The {{< gloss "Session" >}}Session{{< /gloss >}} stays ready the whole time, so reclaiming this capacity costs the user nothing. This page explains what a snapshot captures, when kagent suspends an Actor, and what happens when a suspended Actor is addressed again. ## Actor lifecycle operations @@ -87,6 +87,6 @@ Because kagent suspends at every turn boundary, every message to a Session pays A **checkpoint** marks one point in a conversation and keeps it. You can create a checkpoint when a Session reaches a state worth returning to, such as a long investigation that is only partway through. You can then start a second Session from that point, which is known as a {{< gloss "Fork" >}}fork{{< /gloss >}}. The fork continues the conversation from where the checkpoint marked it, new turns append only to the fork, and the fork gets its own conversation and task identifiers rather than inheriting the source's. To create a checkpoint and fork a Session from it, follow the [Agent Substrate example]({{< link path="examples/agent-substrate" >}}). -Checkpoint a Session only at a turn boundary, because a checkpoint captures this boundary. The request names the terminal task to pin, and it fails when a newer turn has already started or while this turn's snapshot is still being written. +Create a checkpoint only after a turn finishes. The request names the task that the turn ended on, and it fails when a newer turn has already started or while that turn's snapshot is still being written. Underneath, a checkpoint pins a snapshot that would otherwise be discarded. Agent Substrate treats the snapshot written on each suspend as transient and collects it once a newer one supersedes it. Creating a checkpoint attaches a {{< gloss "Tag" >}}Tag{{< /gloss >}} to the snapshot that the Session most recently suspended to, and Agent Substrate keeps any snapshot that a tag still names. Deleting the checkpoint removes the tag and releases the snapshot. diff --git a/docs-site/content/kagent/1.x/operations/operational-considerations.md b/docs-site/content/kagent/1.x/operations/operational-considerations.md index d9f557b7..e5bb27e2 100644 --- a/docs-site/content/kagent/1.x/operations/operational-considerations.md +++ b/docs-site/content/kagent/1.x/operations/operational-considerations.md @@ -19,16 +19,16 @@ database: key: connectionString ``` -Run your own PostgreSQL instance for production, so that the data outlives the cluster that kagent runs on. +`kagent install` creates a development database in the cluster, so an installation needs no external database to start. Run your own PostgreSQL instance for production, so that the data outlives the cluster that kagent runs on. ### The development database `kagent install` creates a database for you unless you pass `--skip-database-setup`. It deploys a single-replica `kagent-postgresql` Deployment backed by a 500Mi PersistentVolumeClaim, creates the roles that kagent and Agent Substrate connect as, and writes the connection Secrets for both: `kagent-postgres` in the kagent namespace, and `substrate-postgres-readwrite` and `substrate-postgres-owner` in `ate-system`. -Connections to this database carry no password. Each client authenticates with a Pod Certificate projected into its pod, and the connection string points `sslcert` and `sslkey` at the files under `/run/postgres.podcert.ate.dev`. `database.postgres.clientCertificate.enabled` projects those files, and `kagent install` sets it. +Connections to this database carry no password. When `database.postgres.clientCertificate.enabled` is set to `true`, which `kagent install` does, each client authenticates with a Pod Certificate projected into its pod under `/run/postgres.podcert.ate.dev`. The connection string points its `sslcert` and `sslkey` parameters at those files. > [!IMPORTANT] -> The development database is a single pod backed by one PersistentVolumeClaim, and `kagent install` sizes that claim at 500Mi. Losing the pod's volume loses every Session and every compiled revision, so move a production installation to a PostgreSQL instance of your own. +> The development database is a single pod backed by one PersistentVolumeClaim, and `kagent install` sizes that claim at 500Mi. Losing the pod's volume loses every Session and every compiled revision, so move a production installation to a PostgreSQL instance that you manage. ### Use an external PostgreSQL instance @@ -66,7 +66,7 @@ For production, run PostgreSQL outside the cluster's lifecycle so that a kagent | Setting | Description | | ------- | ----------- | | `database.postgres.connectionStringSecretRef` | The Secret and key that hold the connection string. Keeping the string in a Secret keeps the credentials out of Helm values. The controller restarts to pick up a changed Secret. | - | `database.postgres.clientCertificate` | Set `enabled` to `true` only for a database that authenticates clients with Pod Certificates, such as the one that `kagent install` creates. The connection string must then reference the files under `/run/postgres.podcert.ate.dev`. | + | `database.postgres.clientCertificate` | Set `enabled` to `true` only for a database that authenticates clients with Pod Certificates, such as the one that `kagent install` creates. The connection string must then point its `sslcert` and `sslkey` parameters at the files under `/run/postgres.podcert.ate.dev`. An external instance that authenticates with a password, such as the one in the preceding example, leaves `enabled` at `false`. | | `database.postgres.role` | The role that kagent assumes on each connection. Give each installation that shares one database a distinct role. | | `database.postgres.schema` | The schema that holds kagent's tables. `database.postgres.vectorSchema` names the schema that holds the shared `pgvector` extension, and kagent needs `USAGE` on it. | | `database.postgres.vectorEnabled` | Set to `true` only when the instance has the `pgvector` extension installed. The setting enables the vector migration, and features that depend on it, such as [long-term memory]({{< link path="agents/agent-memory" >}}), fail without the extension. The development database does not include `pgvector`. | @@ -102,7 +102,7 @@ Leader election keeps the replicas from conflicting. One replica holds a Kuberne A {{< gloss "Worker" >}}Worker{{< /gloss >}} is an ordinary Kubernetes pod, so every node event reaches the {{< gloss "Actor" >}}Actors{{< /gloss >}} that this Worker hosts. What the event costs depends on what each Actor was doing when the node went away. * A suspended Actor costs nothing. Its state is already written to snapshot storage, and it occupies no Worker while it waits, so the next message resumes it onto whichever Worker in the pool has availability. -* An awake Actor does not survive. Its conversation lives in the memory of its sandbox on the Worker, and Agent Substrate writes nothing to snapshot storage until this Actor suspends. +* An awake Actor is lost. Its conversation lives in the memory of its sandbox on the Worker, and Agent Substrate writes nothing to snapshot storage until this Actor suspends. A lost Actor moves to `ACTOR_STATE_CRASHED`, and Agent Substrate refuses `resume` and `suspend` on an Actor in this state. Through kagent, deleting the Actor and starting a new {{< gloss "Session" >}}Session{{< /gloss >}} is the only available option, so the conversation ends wherever the node ends. @@ -157,11 +157,9 @@ Idle time runs from whichever is later: the Session's creation, or its most rece Work in progress is never deleted. A Session with a running task survives past the interval, and so does one with a turn waiting at `INPUT_REQUIRED` or `AUTH_REQUIRED`. A pending lifecycle operation or a checkpoint being captured also holds a Session open. > [!IMPORTANT] -> **Expiry bounds Sessions, not history.** Deleting an idle Session removes the Session, its shares, its runtime row, and its creation receipt, so `GetSession` returns not-found afterward. The A2A context, its tasks, its event history, and any explicit {{< gloss "Checkpoint" >}}checkpoints{{< /gloss >}} stay in PostgreSQL, and a retained checkpoint outlives the Session it was taken on. kagent has no setting that bounds this retained history, so size a database for a long-running installation with the audit history planned separately. +> **Expiry deletes the Session, not its history.** Deleting an idle Session removes the Session, its shares, its runtime row, and its creation receipt, so `GetSession` returns not-found afterward. The A2A context, its tasks, its event history, and any explicit {{< gloss "Checkpoint" >}}checkpoints{{< /gloss >}} stay in PostgreSQL, and a retained checkpoint outlives the Session it was taken on. kagent has no setting that bounds this retained history, so size a database for a long-running installation with the audit history planned separately. -An expired Session releases its creation request ID, unlike an explicit client deletion, which keeps a tombstone. The same caller can reuse that request ID to start a fresh Session. - - + To watch the sweep, scrape `kagent_session_expired_total`, which counts the Sessions that it removed. Each removal also logs `expired idle session` at debug level with the Session's ID and how long it had been idle. diff --git a/docs-site/content/kagent/1.x/operations/tune-agent-substrate.md b/docs-site/content/kagent/1.x/operations/tune-agent-substrate.md index b07b46a4..87c14f9d 100644 --- a/docs-site/content/kagent/1.x/operations/tune-agent-substrate.md +++ b/docs-site/content/kagent/1.x/operations/tune-agent-substrate.md @@ -84,7 +84,7 @@ A {{< gloss "Worker" >}}Worker{{< /gloss >}} hosts many Actors at the same time, Each Worker reports an Actor capacity when it starts, fixed by the Worker build rather than by a field on the WorkerPool. The gVisor and micro-VM Workers that Agent Substrate ships both report a capacity of 1000 Actors, and no setting raises or lowers it. -Agent Substrate checks this capacity on every placement. It weighs a Worker's spare CPU and memory only when the ActorTemplate states what one Actor needs, and the templates that kagent generates for Sessions state nothing. A pool therefore places a Session's Actor based only on a Worker's available slots, regardless of how loaded the Worker's node already is. Worker resource limits still matter, because they govern how Kubernetes packs Workers onto nodes. +Agent Substrate checks this capacity on every placement. It weighs a Worker's spare CPU and memory only when the ActorTemplate states how much CPU and memory one Actor needs. An ActorTemplate that the kagent controller compiles states no per-Actor CPU or memory, so a pool places a Session's Actor on a Worker's available slots alone, regardless of how loaded the Worker's node already is. Worker resource limits still matter, because they govern how Kubernetes packs Workers onto nodes. Size the pool for the number of turns that run at the same time, not for the number of Agents or Sessions that you have created. A cluster with hundreds of Sessions that are each used occasionally needs far fewer Workers than the session count suggests. @@ -103,7 +103,7 @@ substrateWorkerPool: ### Recognize an undersized pool -Agent Substrate reports pool capacity through its metrics rather than through an error that reaches the caller. Each Worker publishes a state of `idle`, `partial`, `at_capacity`, or `unschedulable`, which together show how much of a pool is in use, and a placement that finds no suitable Worker records a `no_capacity` outcome. Agent Substrate treats this outcome as a normal result rather than a failure, so it carries no error type and raises no alert on its own. Watch the share of Workers at `at_capacity` and the rate of `no_capacity` placements to tell a pool that is merely busy from one that is too small. +Agent Substrate reports pool capacity through its metrics rather than through an error that reaches the caller. Each Worker publishes a state of `idle`, `partial`, `at_capacity`, or `unschedulable`, which together show how much of a pool is in use. A placement that finds no suitable Worker records a `no_capacity` outcome. Agent Substrate treats this outcome as a normal result rather than a failure, so it carries no error type and raises no alert on its own. Watch the share of Workers at `at_capacity` and the rate of `no_capacity` placements to tell a pool that is merely busy from one that is too small. None of those signals reach the caller. A turn that cannot get a Worker waits for one, and a turn that waits too long fails with a generic timeout that names neither capacity nor the pool. @@ -113,7 +113,7 @@ ERROR: Message: actor "ai-01a087c0-1d72-775a-9a32-566acac7b685" request timed out ``` -Because short turns release their capacity quickly, a pool under mild pressure absorbs the load and only sheds the requests that wait past the deadline. The result is intermittent timeouts under concurrency rather than a clean failure. Neither the kagent controller log nor the Agent Substrate API log records a capacity message when this happens, so confirm the diagnosis from the `at_capacity` and `no_capacity` signals and from the pool's `READY` count, then raise the replica count. +Because short turns release their capacity quickly, a pool under mild pressure absorbs the load and only sheds the requests that wait past the deadline. The result is intermittent timeouts under concurrency rather than a clean failure. Neither the kagent controller log nor the Agent Substrate API log records a capacity message when this happens, so confirm the diagnosis from the `at_capacity` and `no_capacity` signals and from the pool's `READY` count. If the pool is under pressure, raise the replica count. ### Set resource requests on Workers @@ -153,7 +153,7 @@ Because `rustfs.enabled` defaults to `true`, the Agent Substrate chart installs ### Move snapshots to an external bucket -1. Turn the bundled store off and select the backend in your Agent Substrate Helm values. Setting `atelet.storageBackend` to `s3` selects the S3 client, and any other value selects Google Cloud Storage. +1. Turn the bundled store off and select the backend in your Agent Substrate Helm values. Setting `atelet.storageBackend` to `s3` selects the S3 client, and setting it to `gcs` selects Google Cloud Storage. Agent Substrate matches only the `s3` value, so every other value, an empty one included, selects Google Cloud Storage. ```yaml rustfs: @@ -164,9 +164,9 @@ Because `rustfs.enabled` defaults to `true`, the Agent Substrate chart installs 2. Supply the credentials that your backend needs. Setting `rustfs.enabled` to `false` stops the chart from setting the `AWS_*` variables, so neither component receives an endpoint or a key from Helm. Each storage client reads its credentials from the pod's own environment instead. - - For Google Cloud Storage, the client uses application default credentials. Bind the components' ServiceAccounts through Workload Identity and set nothing further. + - For Google Cloud Storage, the client uses application default credentials. Bind the components' ServiceAccounts through [Workload Identity Federation for GKE](https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity). The chart renders no Helm value for Google credentials, so set nothing further. - For Amazon S3, the client reads the standard AWS environment and uses a role that the pod assumes. No static key is needed. - - For an S3-compatible store, pass `AWS_ENDPOINT_URL`, `AWS_S3_USE_PATH_STYLE`, and the static credentials through `atelet.extraEnv`. + - For an S3-compatible store, pass two variables and the static credentials through `atelet.extraEnv`. `AWS_ENDPOINT_URL` names the store's API endpoint, which the client otherwise resolves to Amazon S3. `AWS_S3_USE_PATH_STYLE` puts the bucket name in the request path instead of the hostname, which most S3-compatible stores require. ```yaml atelet: @@ -190,11 +190,11 @@ Because `rustfs.enabled` defaults to `true`, the Agent Substrate chart installs location: s3://ate-snapshots/kagent/ ``` -4. Run a turn on an agent, suspend its Actor, and compare the `latestSnapshot` path [in the response](#inspect-the-runtime) against the `location` that you set. Because the backend and the location are configured separately, a mismatch between them breaks resume rather than install. Confirm the result rather than the intent. +4. Run a turn on an agent, suspend its Actor, and compare the `latestSnapshot` path [in the response](#inspect-the-runtime) against the `location` that you set. Because the backend and the location are configured separately, a mismatch between them breaks resuming an Actor from its snapshot later. ### How the chart reaches the components -The `atelet` DaemonSet writes a snapshot, and the `ate-api-server` Deployment manages the snapshot. The chart renders `atelet.storageBackend` as the `ATE_STORAGE_BACKEND` variable on both workloads, so one Helm value keeps both in agreement. +The `atelet` DaemonSet writes a snapshot, and the `ate-api-server` Deployment manages the snapshot. The chart renders `atelet.storageBackend` as the `ATE_STORAGE_BACKEND` variable on the DaemonSet and the Deployment alike, so one Helm value keeps the two in agreement. Reading these variables off a running pod helps when a snapshot fails to write. Edit the Helm values rather than the pod environment because the next Helm upgrade overwrites a direct edit. From b37924da2952eca78e631df7596bc8d5c24fe83c Mon Sep 17 00:00:00 2001 From: Rachael Graham Date: Tue, 13 Oct 2026 10:21:35 -0500 Subject: [PATCH 10/10] bloat pass Signed-off-by: Rachael Graham --- .../1.x/about/architecture/agent-substrate.md | 30 ++++++++-------- .../content/kagent/1.x/about/core-concepts.md | 34 ++++++++----------- .../networking-and-egress.md | 21 +++++------- .../1.x/about/substrate-runtime/sandboxing.md | 4 +-- .../substrate-runtime/suspend-and-resume.md | 8 ++--- .../operations/operational-considerations.md | 16 ++++----- .../1.x/operations/tune-agent-substrate.md | 20 +++++------ 7 files changed, 63 insertions(+), 70 deletions(-) diff --git a/docs-site/content/kagent/1.x/about/architecture/agent-substrate.md b/docs-site/content/kagent/1.x/about/architecture/agent-substrate.md index b98985ea..cf727f3c 100644 --- a/docs-site/content/kagent/1.x/about/architecture/agent-substrate.md +++ b/docs-site/content/kagent/1.x/about/architecture/agent-substrate.md @@ -11,32 +11,32 @@ aliases: ## ActorTemplate -An **ActorTemplate** is the definition that Substrate creates an Actor from. It carries what Substrate needs in order to start this Actor: the agent's container image, the command that starts the agent, the agent's environment, the sandbox class to isolate it with, and the WorkerPool to schedule it onto. The kagent controller produces an ActorTemplate by compiling an {{< gloss "Agent" >}}Agent{{< /gloss >}} and the resources that it names, so an ActorTemplate is where kagent's configuration model becomes executable by Substrate. +An **ActorTemplate** is the definition that Agent Substrate creates an Actor from. It states the agent's container image, the command that starts the agent, the agent's environment, the sandbox class to isolate it with, and the WorkerPool to schedule it onto. The kagent controller compiles an {{< gloss "Agent" >}}Agent{{< /gloss >}} and the resources that it names into an ActorTemplate. -Substrate rejects any change to an ActorTemplate's spec after it is created. This immutability ensures that a running conversation remains on a fixed definition. The controller creates a new ActorTemplate for every compiled {{< gloss "Revision" >}}revision{{< /gloss >}} instead of editing an existing one, and it reclaims an old ActorTemplate once nothing references its revision. An Agent holds a reference for as long as the revision is its desired or its latest successful one, and a Session holds one for as long as it runs on that revision. +Agent Substrate rejects any change to an ActorTemplate's spec after it is created, so a running Session stays on a fixed definition. The controller creates a new ActorTemplate for every compiled {{< gloss "Revision" >}}revision{{< /gloss >}} instead of editing an existing one, and it reclaims an old ActorTemplate once nothing references its revision. An Agent references one for as long as the revision is its desired or its latest successful one, and a Session references one for as long as it runs on that revision. ## Workers and WorkerPools -An Actor needs somewhere to run. Each Actor runs on a **Worker**, a pre-started pod that Substrate places Actors onto. Instead of starting a new pod each time a Session needs an Actor, Substrate schedules this Actor onto a Worker that is already running. The sandbox boundary sits around the Actor, not around the Worker pod that hosts it. +Each Actor runs on a **Worker**, a pod that Agent Substrate starts in advance. A Session that needs an Actor gets one on a Worker that is already running, so no Actor waits for a new pod. The sandbox boundary sits around the Actor, not around the Worker pod that hosts it. ### WorkerPools Workers come from a **WorkerPool**, a Kubernetes custom resource that an operator provisions before any Agent can compile. A WorkerPool declares how many Workers to keep running and which sandbox technology those Workers use. -An operator never creates a Worker directly. Substrate manages them, keeping enough ready in each WorkerPool so that an Actor can start or resume on one immediately, without waiting on the Kubernetes scheduler to place a new Pod. +An operator never creates a Worker directly. Agent Substrate manages them and keeps enough ready in each WorkerPool for an Actor to start or resume immediately. -A WorkerPool exposes a Kubernetes scale subresource. A HorizontalPodAutoscaler can use it to size a pool from a metric instead of a fixed replica count. Agent Substrate publishes how many Workers in each pool have no free Actor slot. An autoscaler that watches this number adds Workers before the pool fills, rather than after placements start to fail. +A WorkerPool exposes a Kubernetes scale subresource, so a HorizontalPodAutoscaler can size a pool from a metric instead of a fixed replica count. Agent Substrate publishes how many Workers in each pool have no free Actor slot. An autoscaler that watches this number adds Workers before the pool fills, instead of after placements fail. ### Worker capacity -One Worker hosts many Actors at the same time. When each Worker starts, it reports a maximum capacity of Actors that it can host. A Worker that reaches this capacity refuses any further Actors, and Substrate puts those Actors on another Worker in the pool. The Worker's Actor capacity is a property of the Worker build, not a setting on the pool. The gVisor and micro-VM Workers that Agent Substrate ships both report a capacity of 1000 Actors. +One Worker hosts many Actors at the same time. Each Worker reports its maximum Actor capacity when it starts. A Worker at that capacity refuses further Actors, and Agent Substrate places them on another Worker in the pool. The Worker's Actor capacity is a property of the Worker build, not a setting on the pool. The gVisor and micro-VM Workers that Agent Substrate ships both report a capacity of 1000 Actors. ### Actor placement -Substrate checks two things before it places an Actor on a Worker: +Agent Substrate checks two things before it places an Actor on a Worker: * Whether the Worker has a free Actor slot. This check always runs. * Whether the Worker has enough spare CPU and memory. This check runs only when the ActorTemplate states how much CPU and memory one Actor needs. @@ -44,7 +44,7 @@ An ActorTemplate that the kagent controller compiles states no per-Actor CPU or -To determine where an Actor is placed, Substrate first narrows the Worker pool to the Workers that can take the Actor. It then picks two of those Workers at random and places the Actor on the less loaded of the two. Substrate determines a Worker's load by picking either the share of its Actor slots that are filled or the share of its CPU and memory that is committed, depending on which is higher. Sampling two Workers rather than ranking the whole pool keeps the decision cheap, and it avoids the pile-up that follows when every placement aims at whichever Worker looks emptiest at that moment. +Agent Substrate first narrows the pool to the Workers that can take the Actor. It then picks two of those Workers at random and places the Actor on the less loaded of the two. A Worker's load is the higher of two shares: the share of its Actor slots that are filled, and the share of its CPU and memory that is committed. Sampling two Workers keeps the decision cheap, and it avoids the pile-up that follows when every placement aims at the Worker that looks emptiest. @@ -54,35 +54,35 @@ An **atespace** is the isolation boundary that an Actor belongs to, and the firs An atespace resembles a Kubernetes namespace, but it is not one. Agent Substrate defines it as its own cluster-wide resource. No namespace contains it, a namespace's RBAC does not reach it, and deleting a Kubernetes namespace does not delete the atespace that kagent named after it. -Actors and Workers sit differently against this boundary. An Actor belongs to exactly one atespace for its entire life, whereas a Worker belongs to none. A Worker hosts whichever Actors Substrate places on it, and those Actors can come from different atespaces. +Actors and Workers sit differently against this boundary. An Actor belongs to exactly one atespace for its entire life, whereas a Worker belongs to none. A Worker hosts whichever Actors Agent Substrate places on it, and those Actors can come from different atespaces. -kagent names each atespace after the Kubernetes namespace of the Agent whose Actor it holds, and creates that atespace on demand the first time a Session in the namespace needs an Actor. The Actor's own name is `session-` followed by the Session's identifier. A Session on an Agent in the `kagent` namespace therefore runs on an Actor that Agent Substrate addresses within the `kagent` atespace. Both halves of this identity appear in the address that traffic uses to reach the Actor, which [Sandboxing]({{< link path="about/substrate-runtime/sandboxing#how-traffic-reaches-a-sandboxed-actor" >}}) covers. +kagent names each atespace after the Kubernetes namespace of the Agent that owns the Actor, and creates the atespace the first time a Session in that namespace needs an Actor. The Actor's own name is `session-` followed by the Session's identifier. A Session on an Agent in the `kagent` namespace therefore runs on an Actor in the `kagent` atespace. Both halves of this identity appear in the address that traffic uses to reach the Actor, which [Sandboxing]({{< link path="about/substrate-runtime/sandboxing#how-traffic-reaches-a-sandboxed-actor" >}}) covers. ## Sandboxing -Because an Actor often runs a model-directed agent that calls tools and executes commands, Substrate runs each Actor in an isolated sandbox rather than a plain container. A WorkerPool's `sandboxClass` field selects the sandbox technology for its Workers: [gVisor](https://gvisor.dev), or a micro-VM that runs the workload under [Cloud Hypervisor](https://www.cloudhypervisor.org) with a [Kata Containers](https://katacontainers.io) kernel and root image. Both technologies isolate an Actor from its Worker's host kernel, and both support suspend and resume operations. +Because an Actor often runs a model-directed agent that calls tools and executes commands, Agent Substrate runs each Actor in an isolated sandbox rather than a plain container. A WorkerPool's `sandboxClass` field selects the sandbox technology for its Workers: [gVisor](https://gvisor.dev), or a micro-VM that runs the workload under [Cloud Hypervisor](https://www.cloudhypervisor.org) with a [Kata Containers](https://katacontainers.io) kernel and root image. Both technologies isolate an Actor from its Worker's host kernel, and both support suspend and resume operations. kagent compiles each ActorTemplate to the sandbox class of the WorkerPool that hosts it, so a default installation runs kagent agents in a {{< gloss "gVisor" >}}gVisor{{< /gloss >}} sandbox, and a pool on the `microvm` class runs them under the micro-VM. A WorkerPool that backs kagent Harnesses must run the Worker build that matches its class. For what each class isolates, see [Sandboxing]({{< link path="about/substrate-runtime/sandboxing" >}}). ## Suspend, snapshot, and resume -Substrate's density model rests on one fact about agent workloads: an Actor spends most of its time idle, waiting on a person or a large language model (LLM) to respond, not actively computing. Substrate exploits that by suspending idle Actors and reclaiming the capacity that they held, then resuming them on demand when traffic arrives. Suspending and resuming allows a WorkerPool to run far more Actors than it has Workers for at any given moment. +An Actor spends most of its time idle, waiting on a person or a large language model (LLM) to respond. Agent Substrate suspends an idle Actor, reclaims the capacity that it held, and resumes the Actor when traffic arrives. A WorkerPool can therefore run far more Actors than it has room for at any one moment. The following diagram traces an Actor through one suspend-and-resume cycle, and shows the second path that opens up once the resulting snapshot is tagged.

{{< reuse "kagent-docs/snippets/snapshot-cycle-diagram.md" >}} -A **WorkerPool** keeps **Workers** running and ready, and a Worker hosts the **Actor** alongside other Actors while its conversation is active. Suspending this Actor writes its full state to an immutable **ActorSnapshot** and releases the capacity that it held on its Worker. +A **WorkerPool** keeps **Workers** running and ready, and a Worker hosts the **Actor** alongside other Actors while its Session is active. Suspending this Actor writes its full state to an immutable **ActorSnapshot** and releases the capacity that it held on its Worker. The diagram forks at this snapshot, because a snapshot serves two purposes. - **Resume** restores the same Actor onto **any Worker in the pool with room**, which is not necessarily the Worker that it ran on before. Because the snapshot captures the Actor's full state, the conversation continues from where it left off. Every idle agent takes this path. - A **{{< gloss "Tag" >}}Tag{{< /gloss >}}** pins this snapshot, and a **New Actor** can be seeded from the tag at the moment that it is created. Resuming an existing Actor never goes through a tag. -A tag gives a snapshot a stable, human-meaningful name, so callers do not need to track Substrate's internal snapshot identity. A tag names one snapshot permanently, and only its visibility scope can change afterward. A tag also acts as a retention pin, so Substrate does not delete a snapshot while a tag still names it. +A tag gives a snapshot a stable name, so callers do not track Agent Substrate's internal snapshot identity. The tag names one snapshot permanently, and only its visibility scope can change afterward. A tag is also a retention pin, so Agent Substrate does not delete a snapshot while a tag still names it. -For example, an agent partway through a long incident investigation reaches a state worth keeping. Creating a [checkpoint]({{< link path="about/substrate-runtime/suspend-and-resume#checkpoints" >}}) tags the snapshot that the agent most recently suspended to, which holds that one snapshot in place while the agent carries on and writes newer ones. Without the tag, Substrate collects this snapshot once a newer one supersedes it. +For example, an agent partway through a long incident investigation reaches a state worth keeping. Creating a [checkpoint]({{< link path="about/substrate-runtime/suspend-and-resume#checkpoints" >}}) tags the snapshot that the agent most recently suspended to, which holds that one snapshot in place while the agent carries on and writes newer ones. Without the tag, Agent Substrate collects this snapshot once a newer one supersedes it. diff --git a/docs-site/content/kagent/1.x/about/core-concepts.md b/docs-site/content/kagent/1.x/about/core-concepts.md index a6ebeb97..158f6cb4 100644 --- a/docs-site/content/kagent/1.x/about/core-concepts.md +++ b/docs-site/content/kagent/1.x/about/core-concepts.md @@ -20,9 +20,9 @@ The new model separates what an agent can do from how it is allowed to run, and - A [**Session**](#session) is a running conversation with an Agent. - An [**Actor**](#actor) is the sandboxed process, provided by Substrate, that a Session runs on. -Agents do not run as Kubernetes Deployments. kagent runs every conversation on [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}), a compute layer that starts capacity in advance and places sandboxed workloads onto it. Substrate keeps a conversation alive without keeping it resident. When a conversation goes idle, Substrate saves its state to a {{< gloss "Snapshot" >}}snapshot{{< /gloss >}} and releases the capacity it held, then restores it from that snapshot when the next message arrives. A caller never sees the pause. +Agents do not run as Kubernetes Deployments. kagent runs every Session on [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}), a compute layer that starts capacity in advance and places sandboxed workloads onto it. An idle Actor is suspended to a {{< gloss "Snapshot" >}}snapshot{{< /gloss >}} and restored when the next message arrives, and a caller never sees the pause. -Substrate needs its own definition of the workload to run, so the kagent controller automatically compiles each Agent into an **{{< gloss "ActorTemplate" >}}ActorTemplate{{< /gloss >}}**, the Substrate resource that an Actor is created from. +Agent Substrate needs its own definition of the workload to run, so the kagent controller compiles each Agent into an **{{< gloss "ActorTemplate" >}}ActorTemplate{{< /gloss >}}**, the Substrate resource that an Actor is created from. The following diagram shows how an Agent becomes a running conversation.

@@ -68,13 +68,9 @@ A **Harness** in kagent is a Kubernetes custom resource that applies that idea t - **Environment**: Literal environment values for the runtime, set in `spec.env`. - **Substrate policy**: The capacity that the agent runs on, and where its state is kept when it goes idle. The Harness names a [WorkerPool]({{< link path="about/architecture/agent-substrate#workers-and-workerpools" >}}) to schedule onto and the storage location that snapshots write to. -A Harness names no AgentTemplate, and an AgentTemplate names no Harness. An [Agent](#agent) pairs the two, and nothing pairs them implicitly. +A Harness names no AgentTemplate, and an AgentTemplate names no Harness. An [Agent](#agent) pairs the two, and nothing pairs them implicitly. Applying a Harness runs nothing on its own. It registers a runtime and policy that an Agent can select. -Each runtime accepts a different subset of configuration, and an Agent that asks for something its runtime cannot do reports a compatibility condition rather than running. For the per-runtime restrictions and the condition that reports them, see the [API reference]({{< link path="reference/api-ref#harness" >}}). - -A Harness owns no running compute by itself. Applying one registers a runtime and policy that an Agent can select. - -For the complete Harness schema, see the [API reference]({{< link path="reference/api-ref#harness" >}}). +Each runtime accepts a different subset of configuration, and an Agent that asks for something its runtime cannot do reports a compatibility condition rather than running. For the per-runtime restrictions and the complete Harness schema, see the [API reference]({{< link path="reference/api-ref#harness" >}}). > [!IMPORTANT] > This `Harness` is unrelated to 0.x's `AgentHarness` resource, which provisions OpenClaw or Hermes coding-agent sandboxes. `Harness` is a different resource that covers how any agent is allowed to run, not a renamed or expanded version of `AgentHarness`. @@ -94,9 +90,9 @@ For the complete AgentTemplate schema, see the [API reference]({{< link path="re ## Agent -An **Agent** is a Kubernetes custom resource that pairs _one AgentTemplate with one Harness_. Each side of the pairing either names an existing resource or carries a complete spec written inline, and an Agent sets exactly one of the two per side. +An **Agent** is a Kubernetes custom resource that pairs _one AgentTemplate with one Harness_. For each side, the Agent either names an existing resource or includes a complete spec inline, and it sets exactly one of the two. -The two sides are set independently. An Agent can name both resources, inline both, or mix the two. An inline spec is a complete definition, not an override of a named resource. kagent creates no Kubernetes object to back one. Every name resolves in the Agent's own namespace, including a name nested inside an inline spec. For the fields that express each choice, see the [API reference]({{< link path="reference/api-ref#agent" >}}). +The two sides are independent, so an Agent can name both resources, inline both, or mix the two. An inline spec is a complete definition, not an override of a named resource, and kagent creates no Kubernetes resource to back one. Every name resolves in the Agent's own namespace, including a name nested inside an inline spec. For the fields that express each choice, see the [API reference]({{< link path="reference/api-ref#agent" >}}). The following Agent resource references both sides: @@ -113,7 +109,7 @@ spec: name: kagent ``` -The Agent owns readiness. Its status carries the desired revision, the latest revision that compiled and prepared successfully, any non-blocking compatibility warnings, and the `Accepted`, `ResolvedRefs`, `Compatible`, and `Ready` conditions. An AgentTemplate and a Harness are shared configuration and carry no runtime status of their own. +The Agent owns readiness. Its status reports the desired revision, the latest revision that compiled and prepared successfully, any non-blocking compatibility warnings, and the `Accepted`, `ResolvedRefs`, `Compatible`, and `Ready` conditions. An AgentTemplate and a Harness are shared configuration and report no runtime status of their own. Deleting an Agent retires its definition. Existing Sessions keep the revisions that they pinned. Recreating an Agent under the same name creates a new identity, which cannot inherit the deleted Agent's last successful revision. @@ -127,7 +123,7 @@ A **Session** is a _running conversation with one Agent_. Unlike the Harness, Ag Each compile produces one **{{< gloss "Revision" >}}revision{{< /gloss >}}**, identified by a digest, which is a SHA-256 hash of the compiled configuration. Because that digest is derived from the configuration itself, editing an Agent or either resource it references compiles to a different digest, and therefore becomes a separate ActorTemplate. kagent never rewrites an existing one. -That immutability keeps running conversations stable. When you create a Session, kagent looks up the Agent's latest successful revision and creates an Actor from it. Editing the Agent afterward does not disturb that Session, which keeps running on the revision that it was created from. Only Sessions created after the edit use the new revision. +That immutability keeps running Sessions stable. When you create a Session, kagent looks up the Agent's latest successful revision and creates an Actor from it. Editing the Agent afterward does not move that Session, so only Sessions created after the edit use the new revision. You can create a Session with the following command: @@ -137,7 +133,7 @@ kagent agent session create --agent assistant -n kagent After it is created, a Session talks to callers over the {{< gloss "A2A" >}}A2A{{< /gloss >}} (Agent-to-Agent) protocol, through kagent's A2A gateway. Callers address the Agent rather than the Session. The HTTP endpoint is `/agents/{namespace}/{name}` and gRPC carries the same `namespace/name` in the standard A2A `tenant` field. The Session's ID is the A2A `contextId`, so a message that carries no context identifier starts a new conversation, and a message that repeats one continues that conversation. -A Session that records no task activity for seven days is deleted by an expiration worker. The `controller.sessionIdleTTL` Helm value sets that window. A value of `0` turns the expiration worker off. For more information about what the deletion retains, see [Expire idle Sessions]({{< link path="operations/operational-considerations#expire-idle-sessions" >}}). +An expiration worker deletes a Session that records no task activity for seven days. The `controller.sessionIdleTTL` Helm value sets that window. A value of `0` turns the expiration worker off. For more information about what the deletion retains, see [Expire idle Sessions]({{< link path="operations/operational-considerations#expire-idle-sessions" >}}). ## Actor @@ -151,30 +147,30 @@ The Harness, AgentTemplate, and Agent resources describe what an agent is. The A ### Compute -An Actor occupies a slot on a Worker, which is a pre-started pod in a WorkerPool. A Worker hosts many Actors at once, and an Actor that goes idle is suspended to a snapshot so that its slot returns to the pool. A pool therefore serves far more Sessions than it holds Workers. Agent Substrate also pauses an Actor to a node-local snapshot, and resumes an Actor onto whichever Worker has room. For these operations and what each snapshot keeps, see [Suspend and resume]({{< link path="about/substrate-runtime/suspend-and-resume" >}}). +An Actor occupies a slot on a Worker, which is a pre-started pod in a WorkerPool. A Worker hosts many Actors at once, and an Actor that goes idle is suspended to a snapshot so that its slot returns to the pool. A pool therefore serves far more Sessions than it has Workers. Agent Substrate also pauses an Actor to a node-local snapshot, and resumes an Actor onto whichever Worker has room. For these operations and what each snapshot keeps, see [Suspend and resume]({{< link path="about/substrate-runtime/suspend-and-resume" >}}). > [!NOTE] > Forking a {{< gloss "Checkpoint" >}}checkpoint{{< /gloss >}} into a second Session is unavailable in this release. kagent captures process memory in every snapshot, and a restored copy would hold the source Session's identifiers, so `ForkSession` rejects the request. Forking returns once Agent Substrate separates the pause and suspend snapshot scopes again. ### Identity -Each Actor carries its own cryptographic identity that is issued as a certificate and scoped to the {{< gloss "Atespace" >}}atespace{{< /gloss >}} that it belongs to. Agent Substrate's egress gateway authorizes every outbound connection against the identity on the certificate rather than against anything in the request. Because of this, an agent cannot reach a destination by claiming to be another agent. +Each Actor has its own cryptographic identity, issued as a certificate and scoped to the {{< gloss "Atespace" >}}atespace{{< /gloss >}} that it belongs to. Agent Substrate's egress gateway authorizes every outbound connection against the identity on the certificate rather than against anything in the request. Because of this, an agent cannot reach a destination by claiming to be another agent. -This identity is also what keeps API keys out of the agent. An agent sends a placeholder header, and the gateway replaces it with the real credential as the request passes through, so the key never enters the agent's environment or its filesystem. For the allowlist that the gateway enforces and how credentials are attached to it, see [Egress control]({{< link path="about/substrate-runtime/networking-and-egress" >}}). +This identity also keeps API keys out of the agent. An agent sends a placeholder header, and the gateway replaces it with the real credential as the request passes through, so the key never enters the agent's environment or its filesystem. For the allowlist that the gateway enforces and how credentials are attached to it, see [Egress control]({{< link path="about/substrate-runtime/networking-and-egress" >}}). ### Isolation -Each Actor runs in its own sandbox, with its own filesystem and its own durable volume for application data. Two Sessions on the same Agent share no files and cannot see any of each other's state, even when they run on the same Worker. One conversation's data therefore stays out of another's context. The sandbox also isolates the workload's system calls from the node's kernel, which matters because an agent runs commands that a model chose. For what each sandbox class separates, see [Sandboxing]({{< link path="about/substrate-runtime/sandboxing" >}}). +Each Actor runs in its own sandbox, with its own filesystem and its own durable volume for application data. Two Sessions on the same Agent share no files and cannot see any of each other's state, even when they run on the same Worker. The sandbox also isolates the workload's system calls from the node's kernel, which matters because an agent runs commands that a model chose. For what each sandbox class separates, see [Sandboxing]({{< link path="about/substrate-runtime/sandboxing" >}}). ## Subagent tools -An agent tool binding can point at another AgentTemplate instead of an MCP server. In this way, a broad-scope agent can hand part of a task to one that is built specifically for that subtask. For example, a release agent delegates a database question to another agent, gets the answer, and carries on. The specialist agent keeps its own prompt and its own tools, and the conversation stays with the broader agent that the caller originally addressed. +An agent tool binding can point at another AgentTemplate instead of an MCP server, so a broad-scope agent can hand part of a task to a specialist. For example, a release agent delegates a database question to another agent, gets the answer, and carries on. The specialist keeps its own prompt and its own tools, and the conversation stays with the agent that the caller addressed. Each subagent binding sets `tools[].subAgent.templateRef` to name an AgentTemplate in the same namespace. The named template compiles under the parent Agent's Harness and runs inside the parent's Actor, so the two agents share one sandbox and the nesting creates no second Actor. A subagent needs no Agent of its own and no matching Harness reference. -Because a subagent runs inside its parent's runtime boundary, the compiler constrains the shape of the resulting tree. Nesting stops at one level. A bound template cannot bind a third. That cap keeps the model predictable, because every agent runs either in its own Actor or in the Actor of the agent that bound it, never deeper. For the remaining rules that a tree must satisfy, see [What a subagent tree allows]({{< link path="skills-and-mcp/about-tools#what-a-subagent-tree-allows" >}}). +Because a subagent runs inside its parent's runtime boundary, nesting stops at one level, and a bound template cannot bind a third. Every agent therefore runs either in its own Actor or in the Actor of the agent that bound it, never deeper. For the remaining rules that a tree must satisfy, see [What a subagent tree allows]({{< link path="skills-and-mcp/about-tools#what-a-subagent-tree-allows" >}}). > [!NOTE] > Dedicated subagents, which would give a bound agent its own Harness, Session, and Actor and reach it over A2A, are not part of the served API. The `subAgent.agentRef` field that would select one is deferred until a dedicated subagent can create and invoke its own Session. diff --git a/docs-site/content/kagent/1.x/about/substrate-runtime/networking-and-egress.md b/docs-site/content/kagent/1.x/about/substrate-runtime/networking-and-egress.md index 37bff87b..eb1376fc 100644 --- a/docs-site/content/kagent/1.x/about/substrate-runtime/networking-and-egress.md +++ b/docs-site/content/kagent/1.x/about/substrate-runtime/networking-and-egress.md @@ -42,9 +42,9 @@ Each rule defines the type of connection that is allowed to a destination and ho - **`https`**: HTTPS, which the gateway intercepts. It terminates the connection with a certificate minted for the destination name, reads the request inside, matches it against the rule's hostnames, and re-originates its own TLS connection onward. HTTP/1.1 and HTTP/2 work inside this connection, and HTTP/3 is not supported. - **`tls_passthrough`**: TLS forwarded to the named servers without decryption. The gateway cannot read the request, so it decides at the TLS ClientHello instead. -All three rule types cover TCP connections. Agent Substrate blocks UDP, and it does not support a protocol in which the server sends the first bytes once the connection opens, such as SMTP or SSH. The gateway reads the destination hostname from the bytes that the client sends first, so those protocols leave the gateway nothing to match against. +All three rule types cover TCP connections. Agent Substrate blocks UDP, and it does not support a protocol in which the server sends the first bytes once the connection opens, such as SMTP or SSH. The gateway reads the destination hostname from the bytes that the client sends first, so those protocols leave the gateway with nothing to match against. -Intercepting HTTPS has a cost. Because the gateway terminates TLS, it holds the cleartext of every agent request for the duration of time it takes to authorize this request. Holding the cleartext also lets the gateway attach credentials on the agent's behalf. For more information, see [Credential injection](#credential-injection). +Intercepting HTTPS has a cost. The gateway decrypts every agent request in order to authorize it, so the cleartext exists at the gateway for the length of that check. Decrypting also lets the gateway attach credentials on the agent's behalf. For more information, see [Credential injection](#credential-injection). ## Rule matching @@ -56,7 +56,7 @@ Every rule names the hosts and ports that it covers, regardless of its protocol. | `ports` | Either every port, or a list of up to 16 port numbers. | | `effects` | Not a match condition. See [Credential injection](#credential-injection). | -A hostname pattern is a DNS name, optionally with a `*` wildcard replacing the complete leftmost label. The wildcard matches exactly one non-empty label, so `*.example.com` matches `api.example.com` but matches neither `example.com` nor `nested.api.example.com`. A `*` wildcard on its own matches every name. No other wildcard syntax is accepted, and a pattern without one matches only the complete name. Patterns must be lowercase, must not carry a trailing dot, and must use the punycode form for an internationalized name. +A hostname pattern is a DNS name, optionally with a `*` wildcard replacing the complete leftmost label. The wildcard matches exactly one non-empty label, so `*.example.com` matches `api.example.com` but matches neither `example.com` nor `nested.api.example.com`. A `*` wildcard on its own matches every name. No other wildcard syntax is accepted, and a pattern without one matches only the complete name. Patterns must be lowercase, must not end with a dot, and must use the punycode form for an internationalized name. ## Policy generation @@ -67,7 +67,7 @@ kagent derives the allowlist from the AgentTemplate rather than taking it as con - Every skill artifact source, such as a git or OCI registry host. - The OpenTelemetry collector, for each telemetry signal that is enabled. -kagent compiles this host set into the Actor's policy when it creates the Actor. Each destination becomes one rule that covers one hostname on one port, and the scheme decides the rule type. For example, an `https` destination becomes an `https` rule, and a cleartext destination becomes an `http` rule. Keeping exactly one rule per host and port is deliberate; otherwise, an overlapping rule could decide a request and skip the credential that the narrower rule carries. A destination that uses both HTTP and HTTPS on the same host and port is rejected rather than compiled. +kagent compiles this host set into the Actor's policy when it creates the Actor. Each destination becomes one rule that covers one hostname on one port, and the scheme decides the rule type. For example, an `https` destination becomes an `https` rule, and a cleartext destination becomes an `http` rule. Keeping exactly one rule per host and port is deliberate. An overlapping rule could otherwise decide a request and skip the credential that the narrower rule attaches. A destination that uses both HTTP and HTTPS on the same host and port is rejected rather than compiled. The host set is fixed when kagent compiles the policy, so an agent reaches only the hosts that its AgentTemplate declares. The gateway denies every other destination, including a web page that a model decides to fetch partway through a Session. @@ -77,14 +77,14 @@ An agent also needs a route to the OpenTelemetry collector, or its traces never ## Policy changes -The gateway caches each Actor's policy rather than fetching it per request, so policy changes are eventually consistent rather than immediate. The cache holds an entry for 10 seconds by default. This interval bounds the lag, so a policy that is created, updated, or deleted reaches new requests within one interval. Deleting a policy produces denials rather than an absence of enforcement because the gateway caches the absence too. +The gateway caches each Actor's policy rather than fetching it per request, so policy changes are eventually consistent. The cache holds an entry for 10 seconds by default, so a policy that is created, updated, or deleted reaches new requests within that window. Deleting a policy produces denials rather than an absence of enforcement, because the gateway caches the absence too. > [!IMPORTANT] -> A Session pins the revision it was created from, so editing an AgentTemplate does not move an existing conversation onto a new allowlist. A Session keeps the destinations that it started with for however long it runs. To pick up a changed set of destinations, create a new Session. +> A Session pins the revision it was created from, so editing an AgentTemplate does not move an existing Session onto a new allowlist. A Session keeps the destinations that it started with for however long it runs. To pick up a changed set of destinations, create a new Session. ## Diagnose a denied request -A denied request fails without explaining itself. The agent sees a rejected call, and its own error text names neither the destination nor the rule that refused it. The gateway records the reason in its log. +A denied request gives no explanation. The agent sees a rejected call, and its own error text names neither the destination nor the rule that refused it. The gateway records the reason in its log. ```bash kubectl logs -n ate-system deploy/atenet-egress -c agentgateway --tail=100 @@ -98,7 +98,7 @@ error request ... http.host=kagent-tools.kagent http.status=403 error="actor egress policy denied: ... \"EgressPolicy not found\"" reason=Authorization ``` -Read this entry in three parts. `reason=Authorization` means that the policy refused the request rather than the request failing due to some other cause. The `error` text names which check failed, such as a policy that does not exist. The `http.host` names the destination that the agent asked for. Compare this host against the AgentTemplate. +Read this entry in three parts. `reason=Authorization` means that the policy refused the request rather than the request failing for another reason. The `error` text names which check failed, such as a policy that does not exist. The `http.host` names the destination that the agent asked for. Compare this host against the AgentTemplate. A `503` means that the gateway could not reach the control plane or a credential provider, which is an installation problem rather than a denied request. @@ -112,10 +112,7 @@ A model provider or an MCP server usually requires an API key or token. The agen kagent sets this up from two resources. Every ModelConfig that holds an `apiKeySecret` becomes a credential on the rule for that model's host, and every RemoteMCPServer whose `headersFrom` reads a Secret becomes a credential on the rule for its host. kagent attaches the credential to the destination's own rule rather than adding a rule for it, and it refuses to compile a credential whose host is not already an allowed destination. -A credential is declared on a rule as a `replace_headers` entry. The entry names the header to replace, an optional prefix such as `Bearer `, and either a reference that a registered credential provider resolves or a Substrate-issued JWT asserting the Actor's identity. - -> [!IMPORTANT] -> The gateway replaces a header only when the request already carries it. A request that omits the placeholder header is forwarded unchanged, without the credential, rather than being denied. +A credential is declared on a rule as a `replace_headers` entry. The entry names the header to replace, an optional prefix such as `Bearer `, and either a reference that a registered credential provider resolves or a JWT that Agent Substrate issues to assert the Actor's identity. The gateway replaces the header only when the request already sends it. A request that omits the placeholder header is forwarded unchanged, without the credential, rather than being denied. > [!NOTE] > Replacement happens only on a connection that the gateway terminates, and only when a credential provider is configured for the reference. If either is missing, the request reaches the destination without the credential. The request fails closed when replacement is attempted and fails. `403` is returned when the provider does not hold the credential or refuses to release it, and `503` is returned when the provider is unreachable. diff --git a/docs-site/content/kagent/1.x/about/substrate-runtime/sandboxing.md b/docs-site/content/kagent/1.x/about/substrate-runtime/sandboxing.md index 90cbc050..e3eb13e9 100644 --- a/docs-site/content/kagent/1.x/about/substrate-runtime/sandboxing.md +++ b/docs-site/content/kagent/1.x/about/substrate-runtime/sandboxing.md @@ -47,7 +47,7 @@ A **SandboxConfig** is a cluster-scoped resource that describes the runtime file Every ActorTemplate must name a SandboxConfig, and Agent Substrate supplies no name of its own. A template cannot be prepared until the configuration that it names exists in the cluster. Many ActorTemplates can share one SandboxConfig. The runtime version is then pinned in a single place, rather than repeated on every template. -A default installation creates one SandboxConfig, named `gvisor-default`. kagent writes this name onto every ActorTemplate that it generates, so each template carries it explicitly. The configuration looks like the following: +A default installation creates one SandboxConfig, named `gvisor-default`. kagent writes this name onto every ActorTemplate that it generates, so each template names it explicitly. The configuration looks like the following: ```yaml apiVersion: ate.dev/v1alpha1 @@ -103,6 +103,6 @@ Agent Substrate creates a Kubernetes NetworkPolicy for each WorkerPool, selectin That policy governs inbound traffic only. Outbound traffic is governed separately by an egress policy that the gateway enforces on every connection that an Actor opens. That policy is default-deny. An Actor reaches a destination only when a rule allows it, and the gateway denies the connection when no rule does. -kagent derives the egress policy from the {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}}, so an agent carries no egress configuration of its own. The destinations it allows come from what the AgentTemplate already names, such as its model provider and its MCP servers. You author neither policy. Agent Substrate owns the ingress policy, and kagent regenerates the egress policy whenever the AgentTemplate changes. For what the policy holds, how a rule is matched, and how to diagnose a denied request, see [Egress control]({{< link path="about/substrate-runtime/networking-and-egress" >}}). +kagent derives the egress policy from the {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}}, so an agent needs no egress configuration of its own. The destinations it allows come from what the AgentTemplate already names, such as its model provider and its MCP servers. You author neither policy. Agent Substrate owns the ingress policy, and kagent regenerates the egress policy whenever the AgentTemplate changes. For what the policy holds, how a rule is matched, and how to diagnose a denied request, see [Egress control]({{< link path="about/substrate-runtime/networking-and-egress" >}}). diff --git a/docs-site/content/kagent/1.x/about/substrate-runtime/suspend-and-resume.md b/docs-site/content/kagent/1.x/about/substrate-runtime/suspend-and-resume.md index b33b9941..d3c6a6da 100644 --- a/docs-site/content/kagent/1.x/about/substrate-runtime/suspend-and-resume.md +++ b/docs-site/content/kagent/1.x/about/substrate-runtime/suspend-and-resume.md @@ -47,9 +47,9 @@ A **DurableDir volume** is the per-Actor application data surface. Its contents Two kinds of snapshot serve different purposes, and both appear in a normal installation. - The **golden snapshot** is shared by every Actor of one ActorTemplate. Agent Substrate captures it once by booting the workload temporarily when the template is created. Every new Actor of this template starts from this snapshot rather than from a cold start. kagent waits for the golden snapshot before an {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}} becomes ready, and the controller reports `waiting for the ActorTemplate golden snapshot` while the capture runs. -- The **last snapshot** belongs to one Actor. Agent Substrate writes it on every suspend and restores this same Actor from it on the next resume. It holds the files and memory that this one conversation built up, so the conversation continues from where it stopped. +- The **last snapshot** belongs to one Actor. Agent Substrate writes it on every suspend and restores this same Actor from it on the next resume. It keeps the files and memory that this one Session built up, so the Session continues from where it stopped. -Snapshots are persisted to object storage so that Actor state is durable and portable across the cluster. Agent Substrate writes snapshots to Google Cloud Storage or to Amazon Simple Storage Service (S3). The S3 client also reaches S3-compatible stores that accept a custom endpoint and path-style addressing. For the settings that select and configure each client, see [Configure snapshot storage]({{< link path="operations/tune-agent-substrate#configure-snapshot-storage" >}}). A {{< gloss "Harness" >}}Harness{{< /gloss >}} names the location for its Actors' snapshots in its `substrate.snapshotPolicy` section. +Agent Substrate writes every snapshot to object storage, so Actor state is durable and portable across the cluster. The supported backends are Google Cloud Storage and Amazon Simple Storage Service (S3), and the S3 client also reaches S3-compatible stores that accept a custom endpoint and path-style addressing. For the settings that select and configure each client, see [Configure snapshot storage]({{< link path="operations/tune-agent-substrate#configure-snapshot-storage" >}}). A {{< gloss "Harness" >}}Harness{{< /gloss >}} names the location for its Actors' snapshots in its `substrate.snapshotPolicy` section. ```yaml spec: @@ -64,11 +64,11 @@ spec: ## Suspension between turns -kagent does not wait for an Actor to go idle for a long stretch before suspending it. It suspends the Actor at every turn boundary, as soon as the conversation reaches a point where nothing is running. +kagent does not wait for an Actor to go idle for a long stretch before suspending it. It suspends the Actor at every turn boundary, as soon as nothing in the Session is running. A turn reaches such a boundary when its task enters a terminal state, or when the task stops to wait on a person, which is the `INPUT_REQUIRED` and `AUTH_REQUIRED` case. At that point kagent suspends the Actor durably and records the exact snapshot that the suspend produced. -The {{< gloss "Session" >}}Session{{< /gloss >}}'s own state does not change while this happens. It stays `READY` throughout, because suspension is a property of the runtime underneath it rather than of the conversation. A caller that lists Sessions sees a ready conversation whether or not an Actor is currently running for it. +The {{< gloss "Session" >}}Session{{< /gloss >}}'s own state does not change while this happens. It stays `READY` throughout, because suspension is a property of the runtime underneath the Session rather than of the Session itself. A caller that lists Sessions sees a ready Session whether or not an Actor is running for it. > [!NOTE] > Creating a Session does not start an Actor running. The Actor is created as suspended, and the first message addressed to the Session resumes it. diff --git a/docs-site/content/kagent/1.x/operations/operational-considerations.md b/docs-site/content/kagent/1.x/operations/operational-considerations.md index e5bb27e2..f84eb7ba 100644 --- a/docs-site/content/kagent/1.x/operations/operational-considerations.md +++ b/docs-site/content/kagent/1.x/operations/operational-considerations.md @@ -5,11 +5,11 @@ weight: 10 author: kagent.dev --- -A default {{< reuse "kagent-docs/snippets/name-product.md" >}} installation runs one controller replica against the development database that `kagent install` creates, and schedules agents onto any node in the cluster. Each of those defaults works for a trial installation, but all three need a decision before you run {{< reuse "kagent-docs/snippets/name-product.md" >}} in production. This page covers which database to run, how many controller replicas to keep, and how to treat the nodes that host Workers. +A default {{< reuse "kagent-docs/snippets/name-product.md" >}} installation runs one controller replica against the development database that `kagent install` creates, and schedules agents onto any node in the cluster. Each default works for a trial installation, and each needs a decision before you run {{< reuse "kagent-docs/snippets/name-product.md" >}} in production: which database to run, how many controller replicas to keep, and how to treat the nodes that host Workers. ## Choose a database -kagent stores conversations, {{< gloss "Session" >}}Sessions{{< /gloss >}}, and compiled {{< gloss "Revision" >}}revisions{{< /gloss >}} in PostgreSQL. The kagent chart deploys no database of its own. It reads a connection string from a Kubernetes Secret, and `database.postgres.connectionStringSecretRef` names that Secret and the key inside it. +kagent stores {{< gloss "Session" >}}Sessions{{< /gloss >}} and compiled {{< gloss "Revision" >}}revisions{{< /gloss >}} in PostgreSQL. The kagent chart deploys no database of its own. It reads a connection string from a Kubernetes Secret, and `database.postgres.connectionStringSecretRef` names that Secret and the key inside it. ```yaml database: @@ -85,14 +85,14 @@ For production, run PostgreSQL outside the cluster's lifecycle so that a kagent ## Run multiple controller replicas -To ensure that a controller failure does not stop reconciliation, set `controller.replicas` to a number higher than `1`. +So that a controller failure does not stop reconciliation, set `controller.replicas` to a number higher than `1`. ```yaml controller: replicas: 3 ``` -Leader election keeps the replicas from conflicting. One replica holds a Kubernetes lease and performs reconciliation, garbage collection, and scheduled runs. The other replicas stay ready and take over when the leader's lease expires. +Leader election keeps the replicas from conflicting. One replica takes a Kubernetes lease and performs reconciliation, garbage collection, and scheduled runs. The other replicas stay ready and take over when the leader's lease expires. > [!IMPORTANT] > If an installation runs on the development database, extra controller replicas do not make the installation highly available. That database is a single pod backed by one PersistentVolumeClaim, so losing it stops every replica at once. Run an external PostgreSQL instance before counting controller replicas toward availability. @@ -102,13 +102,13 @@ Leader election keeps the replicas from conflicting. One replica holds a Kuberne A {{< gloss "Worker" >}}Worker{{< /gloss >}} is an ordinary Kubernetes pod, so every node event reaches the {{< gloss "Actor" >}}Actors{{< /gloss >}} that this Worker hosts. What the event costs depends on what each Actor was doing when the node went away. * A suspended Actor costs nothing. Its state is already written to snapshot storage, and it occupies no Worker while it waits, so the next message resumes it onto whichever Worker in the pool has availability. -* An awake Actor is lost. Its conversation lives in the memory of its sandbox on the Worker, and Agent Substrate writes nothing to snapshot storage until this Actor suspends. +* An awake Actor is lost. The Session's in-flight state lives in the memory of its sandbox on the Worker, and Agent Substrate writes nothing to snapshot storage until this Actor suspends. -A lost Actor moves to `ACTOR_STATE_CRASHED`, and Agent Substrate refuses `resume` and `suspend` on an Actor in this state. Through kagent, deleting the Actor and starting a new {{< gloss "Session" >}}Session{{< /gloss >}} is the only available option, so the conversation ends wherever the node ends. +A lost Actor moves to `ACTOR_STATE_CRASHED`, and Agent Substrate refuses `resume` and `suspend` on an Actor in this state. Through kagent, deleting the Actor and starting a new {{< gloss "Session" >}}Session{{< /gloss >}} is the only available option, so the Session ends with the node. -Many Actors on a node are suspended at any moment, so losing the node crashes only the ones that are mid-turn. Those Actors still lose their conversations, and a node pool that runs Workers therefore needs stricter rules than the rest of the cluster. A Worker pod that Kubernetes deletes gives its Actors time to suspend. Node auto-upgrade, a spot reclaim, and a pool scale-down do not. +Many Actors on a node are suspended at any moment, so losing the node crashes only the ones that are mid-turn. Those Actors still lose their Sessions, and a node pool that runs Workers therefore needs stricter rules than the rest of the cluster. A Worker pod that Kubernetes deletes gives its Actors time to suspend. Node auto-upgrade, a spot reclaim, and a pool scale-down do not. ### Graceful pod deletion @@ -129,7 +129,7 @@ gcloud container node-pools update "${NODE_POOL}" \ ### Spot and preemptible nodes -A provider reclaims spot and preemptible nodes on its own schedule, and each reclaim crashes every Actor that was awake on the node. A pool on reclaimable capacity therefore trades a lower bill for conversations that end without warning. Use these nodes only where the agents on this pool can afford to lose a turn. +A provider reclaims spot and preemptible nodes on its own schedule, and each reclaim crashes every Actor that was awake on the node. A pool on reclaimable capacity therefore trades a lower bill for Sessions that end without warning. Use these nodes only where the agents on this pool can afford to lose a turn. ### WorkerPool scale-down diff --git a/docs-site/content/kagent/1.x/operations/tune-agent-substrate.md b/docs-site/content/kagent/1.x/operations/tune-agent-substrate.md index 87c14f9d..dad6e7ff 100644 --- a/docs-site/content/kagent/1.x/operations/tune-agent-substrate.md +++ b/docs-site/content/kagent/1.x/operations/tune-agent-substrate.md @@ -7,7 +7,7 @@ author: kagent.dev {{< reuse "kagent-docs/snippets/name-product.md" >}} runs every agent on [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}). A default installation creates one {{< gloss "WorkerPool" >}}WorkerPool{{< /gloss >}} named `kagent-default`, holding a single Worker on the `gvisor` sandbox class, and it writes snapshots to the object store that the Agent Substrate chart bundles. -This single Worker reports capacity for 1000 Actors, so a default pool serves many concurrent {{< gloss "Session" >}}Sessions{{< /gloss >}} before it runs out of slots. Three settings require tuning before the installation can carry real traffic: how many Workers to run, where snapshots are stored durably, and what resource requests the Worker pods carry. This page covers your options for tuning each setting, and steps for moving a pool to the `microvm` sandbox class. +This single Worker reports capacity for 1000 Actors, so a default pool serves many concurrent {{< gloss "Session" >}}Sessions{{< /gloss >}} before it runs out of slots. Three settings still need tuning before the installation takes real traffic: how many Workers to run, where snapshots are stored durably, and what resource requests the Worker pods set. This page covers each setting, and the steps for moving a pool to the `microvm` sandbox class. ## Before you begin @@ -15,13 +15,13 @@ This single Worker reports capacity for 1000 Actors, so a default pool serves ma 2. Confirm that you have administrative access to the cluster, because this guide scales WorkerPools and reads cluster-scoped resources. -3. Install [grpcurl](https://github.com/fullstorydev/grpcurl). `GetSubstrateStatus` has no kubectl or kagent command equivalent, so calling it directly is the only way to get the fuller view of the runtime. +3. Install [grpcurl](https://github.com/fullstorydev/grpcurl). `GetSubstrateStatus` has no kubectl or kagent command equivalent, so calling it directly is the only way to see the full runtime state. ## Inspect the runtime -Read the current state of the runtime before you change it. Three values from this section carry into the rest of this guide: the pool name and its `READY` count, the snapshot path that Actors are writing to, and the Worker image that the pool runs. +Read the current state of the runtime before you change it. The rest of this guide uses three values from this section: the pool name and its `READY` count, the snapshot path that Actors are writing to, and the Worker image that the pool runs. -1. Check the WorkerPool's Worker counts. These counts are the only place that capacity trouble surfaces. +1. Check the WorkerPool's Worker counts. These counts show whether the pool is producing the Workers that you asked for. ```bash kubectl get workerpools -n kagent ``` @@ -74,7 +74,7 @@ Read the current state of the runtime before you change it. Three values from th } ``` -A Worker entry gains an `actorId` only while an Actor occupies it, so the Workers list doubles as a view of which capacity is busy. +A Worker entry gains an `actorId` only while an Actor occupies it, so the Workers list also shows which capacity is busy. ## Size a WorkerPool @@ -139,7 +139,7 @@ The same `template` block also accepts `nodeSelector`, `tolerations`, `priorityC ## Configure snapshot storage -Agent Substrate writes every suspended Actor to an object store, so the bucket holds the conversation state that a resume restores. Two places decide where a snapshot lands: +Agent Substrate writes every suspended Actor to an object store, so the bucket stores the Session state that a resume restores. Two places decide where a snapshot lands: - In the Agent Substrate Helm values, `atelet.storageBackend` selects the storage client and `rustfs.enabled` decides whether the chart supplies credentials for it. These two values cover both the components that write snapshots and the ones that manage them. - In each {{< gloss "Harness" >}}Harness{{< /gloss >}}, `spec.substrate.snapshotPolicy.location`, names the bucket and prefix that its Actors write under. The field is required, and it must point into a bucket that the selected client can reach. @@ -149,7 +149,7 @@ Agent Substrate writes every suspended Actor to an object store, so the bucket h Because `rustfs.enabled` defaults to `true`, the Agent Substrate chart installs an in-cluster object store and points the `atelet` DaemonSet and the `ate-api-server` Deployment at it. The store claims a 1Gi PersistentVolumeClaim, and it accepts an access key and secret key that the chart sets to published defaults. > [!IMPORTANT] -> The bundled store is not durable. Its credentials are shared defaults that every installation receives. Snapshots hold agent conversation state, so [move a production installation to your own bucket](#move-snapshots-to-an-external-bucket) before it carries real traffic. +> The bundled store is not durable. Its credentials are shared defaults that every installation receives. Snapshots store agent Session state, so [move a production installation to your own bucket](#move-snapshots-to-an-external-bucket) before it carries real traffic. ### Move snapshots to an external bucket @@ -214,13 +214,13 @@ Agent Substrate builds a path per Actor under the Harness prefix, so one bucket s3://ate-snapshots/kagent/atespaces//actors//snapshots/ ``` -A {{< gloss "Checkpoint" >}}checkpoint{{< /gloss >}} takes a second shape. kagent creates an Agent Substrate {{< gloss "Tag" >}}Tag{{< /gloss >}} for each checkpoint, and tagging copies the Actor's snapshot into the Tag's own prefix. A Tag holds exactly one snapshot, so its objects sit directly under `tags/` with no `snapshots/` segment. The `` segment is the Tag's `metadata.uid`, so nothing in the path records the `checkpoint-` name that kagent gave the Tag. +A {{< gloss "Checkpoint" >}}checkpoint{{< /gloss >}} uses a different path. kagent creates an Agent Substrate {{< gloss "Tag" >}}Tag{{< /gloss >}} for each checkpoint, and tagging copies the Actor's snapshot into the Tag's own prefix. A Tag names exactly one snapshot, so its objects sit directly under `tags/` with no `snapshots/` segment. The `` segment is the Tag's `metadata.uid`, so nothing in the path records the `checkpoint-` name that kagent gave the Tag. ```console s3://ate-snapshots/kagent/atespaces//tags/ ``` -A backup or lifecycle policy must cover both prefixes, because one scoped to `actors/` omits every checkpoint and understates how much the bucket holds. +A backup or lifecycle policy must cover both prefixes, because one scoped to `actors/` omits every checkpoint and understates how much the bucket stores. ## Keep pools on the sandbox class they serve @@ -269,7 +269,7 @@ One rule governs the change. A pool's class, its Worker image, and the cluster's A default installation creates only the `gvisor` class, because the Agent Substrate chart applies `sandboxconfig-gvisor.yaml` and kagent installs nothing for `microvm`. The `microvm` configuration and its runtime assets are staged out of band by the Agent Substrate installer, such as through `hack/install-microvm-deps.sh --install`. -3. Verify that the pool's Workers came up on the build that you set. The `ateomImage` field in the [runtime response](#inspect-the-runtime) reports the image that each Worker is running, which the WorkerPool resource calls `workerImage`. Compare the two, and confirm in the ouput that the pool reports its full replica count as ready. +3. Verify that the pool's Workers came up on the build that you set. The `ateomImage` field in the [runtime response](#inspect-the-runtime) reports the image that each Worker is running, which the WorkerPool resource calls `workerImage`. Compare the two, and confirm in the output that the pool reports its full replica count as ready. ```bash kubectl get workerpools -n kagent