Rewrite conceptual and operational info

docs-site/assets/kagent-docs/snippets/snapshot-cycle-diagram.md

@@ -3,7 +3,7 @@ flowchart LR
     pool["WorkerPool"] --> worker1["Worker"]
     worker1 -->|hosts| actor["Actor<br>(running)"]
     actor -->|suspend| snapshot["ActorSnapshot<br>(immutable)"]
-    snapshot -->|resume| worker2["Any free Worker<br>in the pool"]
+    snapshot -->|resume| worker2["Any Worker with room<br>in the pool"]
     snapshot -->|pinned by| tag["ActorSnapshotTag<br>(retention pin)"]
     tag -->|seeds| newactor["New Actor"]
 ```

docs-site/content/kagent/1.x/about/architecture/agent-substrate.md

@@ -11,51 +11,78 @@ aliases:
 
 ## ActorTemplate
 
-An **ActorTemplate** is the definition that Substrate creates an Actor from. It carries what Substrate needs in order to start that Actor: the agent's container image, the command that starts the agent, the agent's environment, the sandbox class to isolate it with, and the WorkerPool to schedule it onto. The kagent controller produces an ActorTemplate by compiling an {{< gloss "Agent" >}}Agent{{< /gloss >}} and the resources that it names, so an ActorTemplate is where kagent's configuration model becomes executable by Substrate.
+An **ActorTemplate** is the definition that Agent Substrate creates an Actor from. It states the agent's container image, the command that starts the agent, the agent's environment, the sandbox class to isolate it with, and the WorkerPool to schedule it onto. The kagent controller compiles an {{< gloss "Agent" >}}Agent{{< /gloss >}} and the resources that it names into an ActorTemplate.
 
-Substrate rejects any change to an ActorTemplate's spec after it is created. This immutability ensures that a running conversation remains on a fixed definition. The controller creates a new ActorTemplate for every compiled {{< gloss "Revision" >}}revision{{< /gloss >}} instead of editing an existing one, and it reclaims an old ActorTemplate once nothing references its revision. An Agent holds a reference for as long as the revision is its desired or its latest successful one, and a Session holds one for as long as it runs on that revision.
+Agent Substrate rejects any change to an ActorTemplate's spec after it is created, so a running Session stays on a fixed definition. The controller creates a new ActorTemplate for every compiled {{< gloss "Revision" >}}revision{{< /gloss >}} instead of editing an existing one, and it reclaims an old ActorTemplate once nothing references its revision. An Agent references one for as long as the revision is its desired or its latest successful one, and a Session references one for as long as it runs on that revision.
 
 ## Workers and WorkerPools
 
-An Actor needs somewhere to run. Each Actor runs on a **Worker**, a pre-started pod that waits to receive one. Instead of starting a new pod each time a Session needs an Actor, Substrate schedules that Actor onto a Worker that is already running. The sandbox boundary sits around the Actor, not around the Worker pod that hosts it.
+Each Actor runs on a **Worker**, a pod that Agent Substrate starts in advance. A Session that needs an Actor gets one on a Worker that is already running, so no Actor waits for a new pod. The sandbox boundary sits around the Actor, not around the Worker pod that hosts it.
 
-<!-- REVIEW (runtime reviewer): a Worker hosts at most one Actor at a time today. Confirm whether multiple Actors per Worker ships in the target release before this paragraph describes that model. Confirm the autoscaling mechanism and the metrics that it requires; an HPA with a metrics adapter keyed on assigned-worker count is one proposal, and whether queue-depth scaling is supported is unconfirmed. A diagram of pods, Workers, pools, and Actor placement, with the sandbox boundaries marked, is wanted here once the model is settled. -->
+### WorkerPools
 
 Workers come from a **WorkerPool**, a Kubernetes custom resource that an operator provisions before any Agent can compile. A WorkerPool declares how many Workers to keep running and which sandbox technology those Workers use.
 
-An operator never creates a Worker directly. Substrate manages them, keeping enough ready in each WorkerPool so that an Actor can start or resume on one immediately, without waiting on the Kubernetes scheduler to place a new Pod.
+An operator never creates a Worker directly. Agent Substrate manages them and keeps enough ready in each WorkerPool for an Actor to start or resume immediately.
+
+A WorkerPool exposes a Kubernetes scale subresource, so a HorizontalPodAutoscaler can size a pool from a metric instead of a fixed replica count. Agent Substrate publishes how many Workers in each pool have no free Actor slot. An autoscaler that watches this number adds Workers before the pool fills, instead of after placements fail.
+
+<!-- REVIEW (runtime reviewer): queue-depth scaling is still unconfirmed and is deliberately not described here. Confirm whether it is supported before adding it. -->
+<!-- REVIEW (runtime reviewer): the external metric that drives the autoscaler is named per deployment, so this paragraph describes the signal rather than naming the series. Confirm the supported name before documenting a worked HPA example. -->
+
+### Worker capacity
+
+One Worker hosts many Actors at the same time. Each Worker reports its maximum Actor capacity when it starts. A Worker at that capacity refuses further Actors, and Agent Substrate places them on another Worker in the pool. The Worker's Actor capacity is a property of the Worker build, not a setting on the pool. The gVisor and micro-VM Workers that Agent Substrate ships both report a capacity of 1000 Actors.
+
+### Actor placement
+
+Agent Substrate checks two things before it places an Actor on a Worker:
+* Whether the Worker has a free Actor slot. This check always runs.
+* Whether the Worker has enough spare CPU and memory. This check runs only when the ActorTemplate states how much CPU and memory one Actor needs.
+
+An ActorTemplate that the kagent controller compiles states no per-Actor CPU or memory, so the free Actor slot check is the only one that decides where a Session's Actor lands. [Worker resource limits]({{< link path="operations/tune-agent-substrate#set-resource-requests-on-workers" >}}) still matter, because a Worker with no limits reports its whole node as its own capacity.
+
+<!-- REVIEW (runtime reviewer): on a default installation neither side declares resources, so a Worker keeps accepting Actors up to its build capacity no matter how loaded its node is, and node pressure surfaces as eviction rather than as a placement failure. Confirm that this is intended for GA before this section describes it as the expected operating mode. -->
+
+Agent Substrate first narrows the pool to the Workers that can take the Actor. It then picks two of those Workers at random and places the Actor on the less loaded of the two. A Worker's load is the higher of two shares: the share of its Actor slots that are filled, and the share of its CPU and memory that is committed. Sampling two Workers keeps the decision cheap, and it avoids the pile-up that follows when every placement aims at the Worker that looks emptiest.
+
+<!-- REVIEW (runtime reviewer): a diagram of pods, Workers, pools, and Actor placement, with the sandbox boundaries marked, is wanted in this section. -->
 
 ## Atespaces
 
-An **atespace** is the isolation boundary that an Actor belongs to, and the first half of its identity. Agent Substrate addresses an Actor by its atespace and its name together, so the same Actor name can exist in two atespaces without colliding. Although an atespace resembles a Kubernetes namespace, it is not one. Agent Substrate defines it as its own cluster-wide resource, so no namespace contains it, a namespace's RBAC does not reach it, and deleting a Kubernetes namespace does not delete the atespace that kagent named after it. Actors and Workers sit differently against that boundary: an Actor belongs to exactly one atespace for its entire life, and a Worker belongs to none, because a Worker hosts whichever Actor Substrate places on it.
+An **atespace** is the isolation boundary that an Actor belongs to, and the first half of its identity. Agent Substrate addresses an Actor by its atespace and its name together, so the same Actor name can exist in two atespaces without colliding.
+
+An atespace resembles a Kubernetes namespace, but it is not one. Agent Substrate defines it as its own cluster-wide resource. No namespace contains it, a namespace's RBAC does not reach it, and deleting a Kubernetes namespace does not delete the atespace that kagent named after it.
+
+Actors and Workers sit differently against this boundary. An Actor belongs to exactly one atespace for its entire life, whereas a Worker belongs to none. A Worker hosts whichever Actors Agent Substrate places on it, and those Actors can come from different atespaces.
 
-kagent names each atespace after the Kubernetes namespace of the Agent whose Actor it holds, and creates that atespace on demand the first time a Session in the namespace needs an Actor. The Actor's own name is `session-` followed by the Session's identifier. A Session on an Agent in the `kagent` namespace therefore runs on an Actor that Agent Substrate addresses within the `kagent` atespace. Both halves of that identity appear in the address that traffic uses to reach the Actor, which [Sandboxing]({{< link path="about/substrate-runtime/sandboxing#how-traffic-reaches-a-sandboxed-actor" >}}) covers.
+kagent names each atespace after the Kubernetes namespace of the Agent that owns the Actor, and creates the atespace the first time a Session in that namespace needs an Actor. The Actor's own name is `session-` followed by the Session's identifier. A Session on an Agent in the `kagent` namespace therefore runs on an Actor in the `kagent` atespace. Both halves of this identity appear in the address that traffic uses to reach the Actor, which [Sandboxing]({{< link path="about/substrate-runtime/sandboxing#how-traffic-reaches-a-sandboxed-actor" >}}) covers.
 
 ## Sandboxing
 
-Because an Actor often runs a model-directed agent that calls tools and executes commands, Substrate runs each Actor in an isolated sandbox rather than a plain container. A WorkerPool's `sandboxClass` field selects the sandbox technology for its Workers: [gVisor](https://gvisor.dev), or a micro-VM that runs the workload under [Cloud Hypervisor](https://www.cloudhypervisor.org) with a [Kata Containers](https://katacontainers.io) kernel and root image. Both technologies isolate an Actor from its Worker's host kernel, and both support suspend and resume operations.
+Because an Actor often runs a model-directed agent that calls tools and executes commands, Agent Substrate runs each Actor in an isolated sandbox rather than a plain container. A WorkerPool's `sandboxClass` field selects the sandbox technology for its Workers: [gVisor](https://gvisor.dev), or a micro-VM that runs the workload under [Cloud Hypervisor](https://www.cloudhypervisor.org) with a [Kata Containers](https://katacontainers.io) kernel and root image. Both technologies isolate an Actor from its Worker's host kernel, and both support suspend and resume operations.
 
 kagent compiles each ActorTemplate to the sandbox class of the WorkerPool that hosts it, so a default installation runs kagent agents in a {{< gloss "gVisor" >}}gVisor{{< /gloss >}} sandbox, and a pool on the `microvm` class runs them under the micro-VM. A WorkerPool that backs kagent Harnesses must run the Worker build that matches its class. For what each class isolates, see [Sandboxing]({{< link path="about/substrate-runtime/sandboxing" >}}).
 
 ## Suspend, snapshot, and resume
 
-Substrate's density model rests on one fact about agent workloads: an Actor spends most of its time idle, waiting on a person or a large language model (LLM) to respond, not actively computing. Substrate exploits that by suspending idle Actors and reclaiming their Worker, then resuming them on demand when traffic arrives. Suspending and resuming allows a WorkerPool to run far more Actors than it has Workers for at any given moment.
+An Actor spends most of its time idle, waiting on a person or a large language model (LLM) to respond. Agent Substrate suspends an idle Actor, reclaims the capacity that it held, and resumes the Actor when traffic arrives. A WorkerPool can therefore run far more Actors than it has room for at any one moment.
 
 The following diagram traces an Actor through one suspend-and-resume cycle, and shows the second path that opens up once the resulting snapshot is tagged.
 </br></br>
 
 {{< reuse "kagent-docs/snippets/snapshot-cycle-diagram.md" >}}
 
-A **WorkerPool** keeps **Workers** running and ready, and one Worker hosts the **Actor** while its conversation is active. Suspending that Actor writes its full state to an immutable **ActorSnapshot** and frees the Worker that it was running on.
+A **WorkerPool** keeps **Workers** running and ready, and a Worker hosts the **Actor** alongside other Actors while its Session is active. Suspending this Actor writes its full state to an immutable **ActorSnapshot** and releases the capacity that it held on its Worker.
 
-The diagram forks at that snapshot, because a snapshot serves two purposes.
+The diagram forks at this snapshot, because a snapshot serves two purposes.
 
-- **Resume** restores the same Actor onto **any free Worker in the pool**, which is not necessarily the Worker that it ran on before. Because the snapshot captures the Actor's full state, the conversation continues from where it left off. Every idle agent takes this path.
-- A **{{< gloss "Tag" >}}Tag{{< /gloss >}}** pins that snapshot, and a **New Actor** can be seeded from the tag at the moment that it is created. Resuming an existing Actor never goes through a tag.
+- **Resume** restores the same Actor onto **any Worker in the pool with room**, which is not necessarily the Worker that it ran on before. Because the snapshot captures the Actor's full state, the conversation continues from where it left off. Every idle agent takes this path.
+- A **{{< gloss "Tag" >}}Tag{{< /gloss >}}** pins this snapshot, and a **New Actor** can be seeded from the tag at the moment that it is created. Resuming an existing Actor never goes through a tag.
 
-A tag gives a snapshot a stable, human-meaningful name, so callers do not need to track Substrate's internal snapshot identity. A tag names one snapshot permanently, and only its visibility scope can change afterward. A tag also acts as a retention pin, so Substrate does not delete a snapshot while a tag still names it.
+A tag gives a snapshot a stable name, so callers do not track Agent Substrate's internal snapshot identity. The tag names one snapshot permanently, and only its visibility scope can change afterward. A tag is also a retention pin, so Agent Substrate does not delete a snapshot while a tag still names it.
 
-For example, an agent partway through a long incident investigation reaches a state worth keeping. Creating a [checkpoint]({{< link path="about/substrate-runtime/suspend-and-resume#checkpoints" >}}) tags the snapshot that the agent most recently suspended to, which holds that one snapshot in place while the agent carries on and writes newer ones. Without the tag, Substrate collects that snapshot once a newer one supersedes it.
+For example, an agent partway through a long incident investigation reaches a state worth keeping. Creating a [checkpoint]({{< link path="about/substrate-runtime/suspend-and-resume#checkpoints" >}}) tags the snapshot that the agent most recently suspended to, which holds that one snapshot in place while the agent carries on and writes newer ones. Without the tag, Agent Substrate collects this snapshot once a newer one supersedes it.
 
 <!-- REVIEW (performance reviewer): an earlier draft stated a 100 ms target at the ninety-fifth percentile, measured from traffic arrival to the moment the Actor can receive it. Removed pending confirmation of the approved measurement and whether it applies to kagent. Do not restore it as a latency guarantee. -->