Skip to content

Rewrite conceptual and operational info - #586

Merged
Rachael-Graham merged 12 commits into
mainfrom
product-feedback
Oct 13, 2026
Merged

Rachael-Graham merged 12 commits into
mainfrom
product-feedback

Conversation

@Rachael-Graham

Copy link
Copy Markdown
Contributor

No description provided.

Update Agent Substrate docs to describe Workers hosting multiple Actors, placement and capacity limits, pool scaling metrics, and snapshot storage options.

Signed-off-by: Rachael Graham <rachael.graham@solo.io>
Explain the agentic loop, how sandbox classes and SandboxConfigs are selected, and how requests and network policies flow through the runtime. Add a review note flagging uncertainty about custom egress destinations.

Signed-off-by: Rachael Graham <rachael.graham@solo.io>
Signed-off-by: Rachael Graham <rachael.graham@solo.io>
Signed-off-by: Rachael Graham <rachael.graham@solo.io>
Expand operational docs on database and controller availability, Worker node lifecycle risks, idle Session expiry, and provider key rotation. Also refine proxy guidance and correct the Session expiry link.

Signed-off-by: Rachael Graham <rachael.graham@solo.io>
Signed-off-by: Rachael Graham <rachael.graham@solo.io>
Refine the explanations of migration, Harness and Agent resources, and Session lifecycle. Add an overview of Actor compute, identity, and isolation, with links to the detailed runtime docs.

Signed-off-by: Rachael Graham <rachael.graham@solo.io>
@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

Docs preview

Link Points at
Branch preview The newest push to this branch. Updates in place.
Commit preview b37924d only. Frozen.

Both are uploaded Worker versions and serve no production traffic.

Signed-off-by: Rachael Graham <rachael.graham@solo.io>

Substrate checks two things before it places an Actor on a Worker:
* Whether the Worker has a free Actor slot. This check always runs.
* Whether the Worker has enough spare CPU and memory. This check runs only when the ActorTemplate declares what one Actor needs, and kagent declares nothing for a Session. The free Actor slot check is therefore the only one that decides where a Session's Actor lands. [Worker resource limits]({{< link path="operations/tune-agent-substrate#set-resource-requests-on-workers" >}}) still matter, because a Worker with no limits reports its whole node as its own capacity.

@Nadine2016 Nadine2016 Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and kagent declares nothing for a Session >> not sure how this relates to the sentence before where everything it needs is declared in the ActorTemplate. Also this paragraph is about placing the actor on a worker and it is hard to understand why we suddenly talk about the session

{{< reuse "kagent-docs/snippets/name-product.md" >}} 1.0 replaces 0.x's Deployment-based `Agent` custom resource with a new model built around **Harness**, **AgentTemplate**, **Agent**, and **Session**, running on [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}) instead of the plain Kubernetes Deployments that the 0.x model uses. The 1.0 `Agent` kind shares the 0.x name but not its meaning: it pairs an AgentTemplate with a Harness rather than describing a Deployment. This page defines the vocabulary that the rest of the 1.0 model docs use. If you already have a 0.x installation, you migrate to the new model rather than upgrading into it. No controller converts a 0.x resource, and the two API groups share no conversion, so you author the 1.0 resources yourself and retire the 0.x ones. [Migrate from 0.x]({{< link path="operations/upgrade-from-0x#recreate-your-resources" >}}) maps each 0.x resource onto its 1.0 replacement.
{{< reuse "kagent-docs/snippets/name-product.md" >}} 1.0 replaces 0.x's Deployment-based `Agent` custom resource with a new model built around **Harness**, **AgentTemplate**, **Agent**, and **Session**. The 1.0 `Agent` kind keeps the 0.x name but not its meaning, because it pairs an AgentTemplate with a Harness rather than describing a Deployment. This page defines the vocabulary that the rest of the 1.0 documentation uses.

Moving a 0.x installation to 1.0 is a manual migration, and no upgrade path exists. No controller converts a 0.x resource, and the two API groups share no conversion, so you author the 1.0 resources yourself and retire the 0.x ones. [Migrate from 0.x]({{< link path="operations/upgrade-from-0x#recreate-your-resources" >}}) maps each 0.x resource onto its 1.0 replacement.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe make this a note


Each Worker reports an Actor capacity when it starts, fixed by the Worker build rather than by a field on the WorkerPool. The gVisor and micro-VM Workers that Agent Substrate ships both report a capacity of 1000 Actors, and no setting raises or lowers it.

Agent Substrate checks this capacity on every placement. It weighs a Worker's spare CPU and memory only when the ActorTemplate states what one Actor needs, and the templates that kagent generates for Sessions state nothing. A pool therefore places a Session's Actor based only on a Worker's available slots, regardless of how loaded the Worker's node already is. Worker resource limits still matter, because they govern how Kubernetes packs Workers onto nodes.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comment as earlier

### Recognize an undersized pool

A pool with too few Workers does not report a capacity error anywhere. Turns that cannot get a Worker wait for one, and a turn that waits too long fails with a generic timeout that names neither capacity nor the pool.
Agent Substrate reports pool capacity through its metrics rather than through an error that reaches the caller. Each Worker publishes a state of `idle`, `partial`, `at_capacity`, or `unschedulable`, which together show how much of a pool is in use, and a placement that finds no suitable Worker records a `no_capacity` outcome. Agent Substrate treats this outcome as a normal result rather than a failure, so it carries no error type and raises no alert on its own. Watch the share of Workers at `at_capacity` and the rate of `no_capacity` placements to tell a pool that is merely busy from one that is too small.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Agent Substrate reports pool capacity through its metrics rather than through an error that reaches the caller. Each Worker publishes a state of `idle`, `partial`, `at_capacity`, or `unschedulable`, which together show how much of a pool is in use, and a placement that finds no suitable Worker records a `no_capacity` outcome. Agent Substrate treats this outcome as a normal result rather than a failure, so it carries no error type and raises no alert on its own. Watch the share of Workers at `at_capacity` and the rate of `no_capacity` placements to tell a pool that is merely busy from one that is too small.
Agent Substrate reports pool capacity through its metrics rather than through an error that reaches the caller. Each Worker publishes a state of `idle`, `partial`, `at_capacity`, or `unschedulable`, which together show how much of a pool is in use. A placement that finds no suitable Worker records a `no_capacity` outcome. Agent Substrate treats this outcome as a normal result rather than a failure, so it carries no error type and raises no alert on its own. Watch the share of Workers at `at_capacity` and the rate of `no_capacity` placements to tell a pool that is merely busy from one that is too small.

A pool with too few Workers does not report a capacity error anywhere. Turns that cannot get a Worker wait for one, and a turn that waits too long fails with a generic timeout that names neither capacity nor the pool.
Agent Substrate reports pool capacity through its metrics rather than through an error that reaches the caller. Each Worker publishes a state of `idle`, `partial`, `at_capacity`, or `unschedulable`, which together show how much of a pool is in use, and a placement that finds no suitable Worker records a `no_capacity` outcome. Agent Substrate treats this outcome as a normal result rather than a failure, so it carries no error type and raises no alert on its own. Watch the share of Workers at `at_capacity` and the rate of `no_capacity` placements to tell a pool that is merely busy from one that is too small.

None of those signals reach the caller. A turn that cannot get a Worker waits for one, and a turn that waits too long fails with a generic timeout that names neither capacity nor the pool.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
None of those signals reach the caller. A turn that cannot get a Worker waits for one, and a turn that waits too long fails with a generic timeout that names neither capacity nor the pool.
None of those signals reach the caller. A turn that cannot get a Worker waits for one, and a turn that waits too long fails with a generic timeout.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kept the trailing clause, "that names neither capacity nor the pool." It is the diagnostic point of the sentence: the timeout gives an operator nothing to connect back to pool capacity, which is why the section sends them to the metrics instead.

```

Because short turns release their Worker quickly, a pool under mild pressure absorbs the load and only sheds the requests that wait past the deadline. The result is intermittent timeouts under concurrency rather than a clean failure. Neither the kagent controller log nor the Agent Substrate API log records a capacity message when this happens, so diagnose it from the pool's `READY` count and raise the replica count.
Because short turns release their capacity quickly, a pool under mild pressure absorbs the load and only sheds the requests that wait past the deadline. The result is intermittent timeouts under concurrency rather than a clean failure. Neither the kagent controller log nor the Agent Substrate API log records a capacity message when this happens, so confirm the diagnosis from the `at_capacity` and `no_capacity` signals and from the pool's `READY` count, then raise the replica count.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Because short turns release their capacity quickly, a pool under mild pressure absorbs the load and only sheds the requests that wait past the deadline. The result is intermittent timeouts under concurrency rather than a clean failure. Neither the kagent controller log nor the Agent Substrate API log records a capacity message when this happens, so confirm the diagnosis from the `at_capacity` and `no_capacity` signals and from the pool's `READY` count, then raise the replica count.
Because short turns release their capacity quickly, a pool under mild pressure absorbs the load and only sheds the requests that wait past the deadline. The result is intermittent timeouts under concurrency rather than a clean failure. Neither the kagent controller log nor the Agent Substrate API log records a capacity message when this happens, so confirm the diagnosis from the `at_capacity` and `no_capacity` signals and from the pool's `READY` count. If you notice Worker pressure, raise the replica count.

rustfs:
enabled: false
atelet:
storageBackend: s3

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does that mean, I can set this to test and it will point to google cloud storage?


2. Supply the credentials that your backend needs. Setting `rustfs.enabled` to `false` stops the chart from setting the `AWS_*` variables, so neither component receives an endpoint or a key from Helm. Each storage client reads its credentials from the pod's own environment instead.

- For Google Cloud Storage, the client uses application default credentials. Bind the components' ServiceAccounts through Workload Identity and set nothing further.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just confirming that there would not be any helm settings? if so, maybe we should explicitly say that. and maybe link to google docs for workload identity

- name: AWS_ENDPOINT_URL
value: https://objects.example.com
- name: AWS_S3_USE_PATH_STYLE
value: "true"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should explain what these envs are

location: s3://ate-snapshots/kagent/
```

4. Run a turn on an agent, suspend its Actor, and compare the `latestSnapshot` path [in the response](#inspect-the-runtime) against the `location` that you set. Because the backend and the location are configured separately, a mismatch between them breaks resume rather than install. Confirm the result rather than the intent.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
4. Run a turn on an agent, suspend its Actor, and compare the `latestSnapshot` path [in the response](#inspect-the-runtime) against the `location` that you set. Because the backend and the location are configured separately, a mismatch between them breaks resume rather than install. Confirm the result rather than the intent.
4. Run a turn on an agent, suspend its Actor, and compare the `latestSnapshot` path [in the response](#inspect-the-runtime) against the `location` that you set. Because the backend and the location are configured separately, a mismatch between them breaks resuming an Actor from its snapshot later.


### How the chart reaches the components

The `atelet` DaemonSet writes a snapshot, and the `ate-api-server` Deployment manages the snapshot. The chart renders `atelet.storageBackend` as the `ATE_STORAGE_BACKEND` variable on both workloads, so one Helm value keeps both in agreement.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is both referring to here?


- `database.postgres.bundled.enabled` determines whether the chart deploys the bundled PostgreSQL pod and its PersistentVolumeClaim (PVC). This setting does not affect which database the controller connects to.
- `database.postgres.url` and `database.postgres.urlFile` determine what the controller connects to. When either is set, the controller uses it. When both are empty, the controller connects to the bundled instance.
Run your own PostgreSQL instance for production, so that the data outlives the cluster that kagent runs on.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sounds like you need to install an instance in any case?

`kagent install` creates a database for you unless you pass `--skip-database-setup`. It deploys a single-replica `kagent-postgresql` Deployment backed by a 500Mi PersistentVolumeClaim, creates the roles that kagent and Agent Substrate connect as, and writes the connection Secrets for both: `kagent-postgres` in the kagent namespace, and `substrate-postgres-readwrite` and `substrate-postgres-owner` in `ate-system`.

The last row fails at template time with `No database connection configured`, rather than installing a controller that cannot start.
Connections to this database carry no password. Each client authenticates with a Pod Certificate projected into its pod, and the connection string points `sslcert` and `sslkey` at the files under `/run/postgres.podcert.ate.dev`. `database.postgres.clientCertificate.enabled` projects those files, and `kagent install` sets it.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the last sentence should say somethign like When you enable client certificates with the ... setting, ....

right now, it is hard to understand where sslcert and sslkey are coming from. I am assuming those are sub fields under that setting, but not sure.


The bundled instance claims its PersistentVolumeClaim from the cluster's default StorageClass. Set `database.postgres.bundled.storageClassName` to choose a different one.
> [!IMPORTANT]
> The development database is a single pod backed by one PersistentVolumeClaim, and `kagent install` sizes that claim at 500Mi. Losing the pod's volume loses every Session and every compiled revision, so move a production installation to a PostgreSQL instance of your own.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
> The development database is a single pod backed by one PersistentVolumeClaim, and `kagent install` sizes that claim at 500Mi. Losing the pod's volume loses every Session and every compiled revision, so move a production installation to a PostgreSQL instance of your own.
> The development database is a single pod backed by one PersistentVolumeClaim, and `kagent install` sizes that claim at 500Mi. Losing the pod's volume loses every Session and every compiled revision, so move a production installation to a PostgreSQL instance that you manage.

| `database.postgres.url` or `database.postgres.urlFile` | The connection string, or the path to a file holding it. `urlFile` keeps the credentials out of Helm values. |
| `database.postgres.vectorEnabled` | Set to `true` only when the instance has the `pgvector` extension installed. The setting enables the vector migration, and features that depend on it, such as [long-term memory]({{< link path="agents/agent-memory" >}}), fail without the extension. The bundled image does not include `pgvector`. |
| `database.postgres.connectionStringSecretRef` | The Secret and key that hold the connection string. Keeping the string in a Secret keeps the credentials out of Helm values. The controller restarts to pick up a changed Secret. |
| `database.postgres.clientCertificate` | Set `enabled` to `true` only for a database that authenticates clients with Pod Certificates, such as the one that `kagent install` creates. The connection string must then reference the files under `/run/postgres.podcert.ate.dev`. |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we say the connection string must reference the files, but this is not true in the connection string above

A {{< gloss "Worker" >}}Worker{{< /gloss >}} is an ordinary Kubernetes pod, so every node event reaches the {{< gloss "Actor" >}}Actors{{< /gloss >}} that this Worker hosts. What the event costs depends on what each Actor was doing when the node went away.

* A suspended Actor costs nothing. Its state is already written to snapshot storage, and it occupies no Worker while it waits, so the next message resumes it onto whichever Worker in the pool has availability.
* An awake Actor does not survive. Its conversation lives in the memory of its sandbox on the Worker, and Agent Substrate writes nothing to snapshot storage until this Actor suspends.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe a different word than survive. it is hard to understand what this means

When `providers.<provider>.apiKey` is empty and `apiKeySecretRef` names a Secret that you create, such as one that an ExternalSecret syncs, the chart does not render the Secret. Update the Secret wherever you manage it.
{{% /tab %}}
{{% tab name="Helm manages the Secret" %}}
When you set `providers.<provider>.apiKey` in your Helm values, the chart creates the Secret and Helm owns it. Rotate the key through Helm rather than by editing the Secret, because a direct edit does not last. With Helm 3, the next `helm upgrade --reuse-values` writes the previous key back. With Helm 4, which applies changes on the server side, the next upgrade fails with a field ownership conflict on the Secret.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thinking you can remove the helm3/4 sentences as those refer to when you edit the secret directly, which is not what we want them to do

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping these. The review feedback this PR addresses asks specifically to move the providers.<provider>.apiKey, Helm ownership, and upgrade-conflict explanation into a second "Helm manages the Secret" tab, which is where they now sit. They tell a reader who already set the key in Helm values why a direct Secret edit will not hold.


> [!IMPORTANT]
> **Expiry bounds conversations, not history.** Deleting an idle Session removes the Session, its shares, its runtime row, and its creation receipt, and `GetSession` then returns not-found. The A2A context, its tasks, its event history, and any explicit {{< gloss "Checkpoint" >}}checkpoints{{< /gloss >}} stay in PostgreSQL, and a retained checkpoint can still be forked after the Session that it was taken on has expired. Sizing a database for a long-running installation means planning for that audit history separately, because no setting on this page bounds it.
> **Expiry bounds Sessions, not history.** Deleting an idle Session removes the Session, its shares, its runtime row, and its creation receipt, so `GetSession` returns not-found afterward. The A2A context, its tasks, its event history, and any explicit {{< gloss "Checkpoint" >}}checkpoints{{< /gloss >}} stay in PostgreSQL, and a retained checkpoint outlives the Session it was taken on. kagent has no setting that bounds this retained history, so size a database for a long-running installation with the audit history planned separately.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expiry bounds Sessions, not history > needs another title as we are not talking about expiration, but deletion.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retitled to "Expiry deletes the Session, not its history." Keeping expiry in the label, because expiry is what triggers the deletion and the section is about the idle TTL, but the callout now names deletion as you asked.

> **Expiry bounds conversations, not history.** Deleting an idle Session removes the Session, its shares, its runtime row, and its creation receipt, and `GetSession` then returns not-found. The A2A context, its tasks, its event history, and any explicit {{< gloss "Checkpoint" >}}checkpoints{{< /gloss >}} stay in PostgreSQL, and a retained checkpoint can still be forked after the Session that it was taken on has expired. Sizing a database for a long-running installation means planning for that audit history separately, because no setting on this page bounds it.
> **Expiry bounds Sessions, not history.** Deleting an idle Session removes the Session, its shares, its runtime row, and its creation receipt, so `GetSession` returns not-found afterward. The A2A context, its tasks, its event history, and any explicit {{< gloss "Checkpoint" >}}checkpoints{{< /gloss >}} stay in PostgreSQL, and a retained checkpoint outlives the Session it was taken on. kagent has no setting that bounds this retained history, so size a database for a long-running installation with the audit history planned separately.

An expired Session releases its creation request ID, unlike an explicit client deletion, which keeps a tombstone. The same caller can reuse that request ID to start a fresh Session.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need a different first sentence

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the paragraph rather than rewriting it. The review feedback asks to move the creation-request-ID behavior to a more suitable topic, so I folded the fact into the existing REVIEW comment until a Sessions reference page exists to hold it.

---

An agent spends most of its life waiting. It waits on a person to reply, and it waits on a large language model (LLM) to answer. [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}) runs each agent inside an **Actor**, the unit that it suspends and resumes, and it treats that idle time as reclaimable: it suspends an idle Actor into a {{< gloss "Snapshot" >}}snapshot{{< /gloss >}}, frees the {{< gloss "Worker" >}}Worker{{< /gloss >}} that the Actor was running on, and restores the Actor when traffic arrives for it. This page explains what a snapshot captures, when kagent suspends an Actor, and what happens when a suspended Actor is addressed again.
An agent spends most of its life waiting. It waits on a person to reply, and it waits on a large language model (LLM) to answer. [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}) runs each agent inside an **Actor**, the unit that it suspends and resumes. Rather than hold compute open through that waiting, Agent Substrate writes the idle Actor to a {{< gloss "Snapshot" >}}snapshot{{< /gloss >}} and releases the capacity that the Actor held on its {{< gloss "Worker" >}}Worker{{< /gloss >}}. The next message restores it. The {{< gloss "Session" >}}Session{{< /gloss >}} stays ready the whole time, so reclaiming this capacity costs the user nothing. This page explains what a snapshot captures, when kagent suspends an Actor, and what happens when a suspended Actor is addressed again.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
An agent spends most of its life waiting. It waits on a person to reply, and it waits on a large language model (LLM) to answer. [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}) runs each agent inside an **Actor**, the unit that it suspends and resumes. Rather than hold compute open through that waiting, Agent Substrate writes the idle Actor to a {{< gloss "Snapshot" >}}snapshot{{< /gloss >}} and releases the capacity that the Actor held on its {{< gloss "Worker" >}}Worker{{< /gloss >}}. The next message restores it. The {{< gloss "Session" >}}Session{{< /gloss >}} stays ready the whole time, so reclaiming this capacity costs the user nothing. This page explains what a snapshot captures, when kagent suspends an Actor, and what happens when a suspended Actor is addressed again.
An agent spends most of its life waiting. It waits on a person to reply, and it waits on a large language model (LLM) to answer. [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}) runs each agent inside an **Actor**, the unit that suspends and resumes the agent. To free up compute resources while waiting for an agent to resume, Agent Substrate writes the idle Actor to a {{< gloss "Snapshot" >}}snapshot{{< /gloss >}} and releases the capacity that the Actor held on its {{< gloss "Worker" >}}Worker{{< /gloss >}}. The next message to the agent restores the Actor. The {{< gloss "Session" >}}Session{{< /gloss >}} stays ready the whole time, so reclaiming this capacity costs the user nothing. This page explains what a snapshot captures, when kagent suspends an Actor, and what happens when a suspended Actor is addressed again.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Took this with one change: "the unit that Agent Substrate suspends and resumes" rather than "the unit that suspends and resumes the agent." Agent Substrate performs the suspend, not the Actor.

A **checkpoint** marks one point in a conversation and keeps it. You can create a checkpoint when a Session reaches a state worth returning to, such as a long investigation that is only partway through. You can then start a second Session from that point, which is known as a {{< gloss "Fork" >}}fork{{< /gloss >}}. The fork continues the conversation from where the checkpoint marked it, new turns append only to the fork, and the fork gets its own conversation and task identifiers rather than inheriting the source's. To create a checkpoint and fork a Session from it, follow the [Agent Substrate example]({{< link path="examples/agent-substrate" >}}).

A Session must be at a turn boundary to be checkpointed, because the turn boundary is captured. The request names the terminal task to pin. It fails when a newer turn has already started, and again while that turn's snapshot is still being written.
Checkpoint a Session only at a turn boundary, because a checkpoint captures this boundary. The request names the terminal task to pin, and it fails when a newer turn has already started or while this turn's snapshot is still being written.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand the first sentence

## How the gateway authorizes a connection

Outbound traffic leaves a Worker through the egress gateway rather than directly out. That single chokepoint gives Agent Substrate one place to identify the caller and apply rules, and the gateway does both on every connection.
Outbound traffic leaves a Worker through the egress gateway rather than directly out. This chokepoint gives Agent Substrate one place to decide what each agent can reach. The gateway takes three steps on every connection, and each step depends on the one before it, because the Actor's identity selects the policy, and the policy supplies the rules.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Outbound traffic leaves a Worker through the egress gateway rather than directly out. This chokepoint gives Agent Substrate one place to decide what each agent can reach. The gateway takes three steps on every connection, and each step depends on the one before it, because the Actor's identity selects the policy, and the policy supplies the rules.
All outbound traffic leaves a Worker through the egress gateway, which gives Agent Substrate one place to decide what each agent can reach. The gateway takes three steps on every connection, and each step depends on the one before it, because the Actor's identity selects the policy, and the policy supplies the rules.


1. The gateway identifies the Actor. It reads the Actor's SPIFFE ID from the client certificate on the connection from the Actor, and it verifies this certificate rather than trusting anything in the request.

2. The gateway fetches the `EgressPolicy` for this Actor. Each Actor has at most one `EgressPolicy` named `default`, and Agent Substrate addresses the policy by the Actor rather than by this name. An Actor with no policy reaches nothing, because a policy is the only thing that allows a destination.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
2. The gateway fetches the `EgressPolicy` for this Actor. Each Actor has at most one `EgressPolicy` named `default`, and Agent Substrate addresses the policy by the Actor rather than by this name. An Actor with no policy reaches nothing, because a policy is the only thing that allows a destination.
2. The gateway fetches the `EgressPolicy` for this Actor. Each Actor has at most one `EgressPolicy` named `default`, and Agent Substrate addresses the policy by the Actor rather than by this name. An Actor with no policy reaches nothing, because a policy is the only thing that allows access to a destination.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Took your "access to a destination" fix but changed the subject. The review feedback asks to name the gateway as the thing that denies, because a default-deny policy does not stop an Actor from attempting a request, so the sentence now reads: "The gateway denies every request from an Actor that has no policy, because only a policy allows access to a destination."

The gateway then authorizes the request against the Actor's `EgressPolicy`. An Actor has at most one, named `default`. The policy holds an ordered list of rules, and the gateway takes the **first** rule that matches: only that rule applies, and evaluation stops even when a later rule would also match. **A request that matches no rule is denied, and an Actor with no policy at all gets no outbound connection.** A rule can match only what the gateway can read of the request.
A request that no rule matches is denied, and only the rule that applies contributes a credential.

The verified identity holds the other two steps together. The Actor's identity is established only by the certificate, so the gateway authorizes every request against the Actor that actually sent it regardless of the headers or hostnames that the request carries. The gateway refuses a connection that does not present a certificate rather than falling back to a default identity. Each Actor also gets its own connection pool, even when two of them dial the same address, so that every connection carries requests from the one Actor that opened it.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The verified identity holds the other two steps together. The Actor's identity is established only by the certificate, so the gateway authorizes every request against the Actor that actually sent it regardless of the headers or hostnames that the request carries. The gateway refuses a connection that does not present a certificate rather than falling back to a default identity. Each Actor also gets its own connection pool, even when two of them dial the same address, so that every connection carries requests from the one Actor that opened it.
The Actor's identity is established only by the certificate, so the gateway authorizes every request against the Actor that actually sent it regardless of the headers or hostnames that the request carries. The gateway refuses a connection that does not present a certificate rather than falling back to a default identity. Each Actor also gets its own connection pool, even when two of them dial the same address, so that every connection carries requests from the one Actor that opened it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Took the body of this but kept a lead sentence. The review feedback asks the page to explain why the verified Actor identity matters for policy enforcement, so the paragraph now opens with "Policy enforcement depends on this verified identity."

### Traffic types

The gateway allows or denies a request on one of two things: the `Host` header of the request, or the address the Actor dialed. Which of the two it uses depends on whether the traffic carries a request that the gateway can read. That difference splits egress into three cases worth keeping apart.
A rule's type fixes the kind of connection it covers and the place that the gateway reads the destination hostname from. Three types cover the three cases.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
A rule's type fixes the kind of connection it covers and the place that the gateway reads the destination hostname from. Three types cover the three cases.
Each rule defines the type of connection that is allowed to a destination and how the gateway determines the destination's hostname. The following connection types are allowed:

- **Cleartext HTTP**: The gateway reads the `Host` header, so hostname rules apply and a request that matches no rule is refused.
- **TLS**: The gateway terminates the connection with a certificate minted for the destination name, reads the request inside, and matches it against hostname rules exactly as it does cleartext. It then re-originates its own TLS connection to the destination. An allowed request appears in the gateway log with the `http.host` and `http.path` it was matched on.
- **Opaque TCP**: A tunnel that carries no HTTP request gives the gateway nothing to read, so the gateway allows or denies it by address, at its outer hop. Express those destinations as a `cidrs` rule, because a hostname rule has nothing to match against.
- **`http`**: Cleartext HTTP, meaning HTTP/1.1 and the h2c that gRPC uses. The gateway reads the `Host` header and matches it against the rule's hostnames. TLS to those same names is not allowed by this rule.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- **`http`**: Cleartext HTTP, meaning HTTP/1.1 and the h2c that gRPC uses. The gateway reads the `Host` header and matches it against the rule's hostnames. TLS to those same names is not allowed by this rule.
- **`http`**: Cleartext HTTP, meaning HTTP/1.1 and the h2c that gRPC uses. The gateway reads the hostname from the `Host` header and matches it against the rule's hostnames. TLS to those same names is not allowed by this rule.

- **`tls_passthrough`**: TLS forwarded to the named servers without decryption. The gateway cannot read the request, so it decides at the TLS ClientHello instead.

Enforcing hostname rules against encrypted traffic has a cost: because the gateway terminates TLS, it holds the cleartext of every agent request for as long as it takes to authorize it. Holding the cleartext also lets the gateway attach credentials on the agent's behalf. For more information, see [Credential injection](#credential-injection).
All three rule types cover TCP connections. Agent Substrate blocks UDP, and it does not support a protocol in which the server sends the first bytes once the connection opens, such as SMTP or SSH. The gateway reads the destination hostname from the bytes that the client sends first, so those protocols leave the gateway nothing to match against.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
All three rule types cover TCP connections. Agent Substrate blocks UDP, and it does not support a protocol in which the server sends the first bytes once the connection opens, such as SMTP or SSH. The gateway reads the destination hostname from the bytes that the client sends first, so those protocols leave the gateway nothing to match against.
All three rule types cover TCP connections. Agent Substrate blocks UDP, and it does not support a protocol in which the server sends the first bytes once the connection opens, such as SMTP or SSH. The gateway reads the destination hostname from the bytes that the client sends first, so those protocols leave the gateway with nothing to match against.

Signed-off-by: Rachael Graham <rachael.graham@solo.io>
Signed-off-by: Rachael Graham <rachael.graham@solo.io>
@Rachael-Graham
Rachael-Graham merged commit 2199ef7 into main Oct 13, 2026
4 checks passed
@Rachael-Graham
Rachael-Graham deleted the product-feedback branch October 13, 2026 15:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants