From ff6582f9c717999c710df36f4635105e967a1654 Mon Sep 17 00:00:00 2001 From: GitHub Action Date: Mon, 12 Oct 2026 17:53:43 +0000 Subject: [PATCH 1/2] [Automated] Draft docs (kagent): feat: wake session quiescence workers after task settlement Signed-off-by: GitHub Action Signed-off-by: Rachael Graham --- .../1.x/operations/operational-considerations.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs-site/content/kagent/1.x/operations/operational-considerations.md b/docs-site/content/kagent/1.x/operations/operational-considerations.md index d9f557b7..ecc43322 100644 --- a/docs-site/content/kagent/1.x/operations/operational-considerations.md +++ b/docs-site/content/kagent/1.x/operations/operational-considerations.md @@ -97,6 +97,21 @@ Leader election keeps the replicas from conflicting. One replica holds a Kuberne > [!IMPORTANT] > If an installation runs on the development database, extra controller replicas do not make the installation highly available. That database is a single pod backed by one PersistentVolumeClaim, so losing it stops every replica at once. Run an external PostgreSQL instance before counting controller replicas toward availability. +## Tune session quiescence recovery {#session-quiescence-recovery} + +When a task settles, the controller commits the task history to PostgreSQL. Then the controller wakes local workers that pause or suspend the Session's {{< gloss "Actor" >}}Actor{{< /gloss >}}. Committed settlements do not wait for the recovery scan interval. Four workers drain durable claims, and PostgreSQL coordinates the work across controller replicas so that one replica handles each claim. + +If a controller exits after the commit and before the local wake-up, another replica finds the durable claim on its next recovery scan. The default recovery interval is `1m`. Increasing the interval can delay pause, suspension, and {{< gloss "Snapshot" >}}snapshot{{< /gloss >}} availability after a missed wake-up. With a shorter interval, the controller scans for missed work more often. + +Use `KAGENT_SESSION_QUIESCENCE_POLL_INTERVAL` to change only the missed-wake-up recovery interval. The value must be a positive Go duration. The controller retries temporarily blocked work and database errors within one second, regardless of the recovery interval. + +```yaml +controller: + env: + - name: KAGENT_SESSION_QUIESCENCE_POLL_INTERVAL + value: 30s +``` + ## Node operations A {{< gloss "Worker" >}}Worker{{< /gloss >}} is an ordinary Kubernetes pod, so every node event reaches the {{< gloss "Actor" >}}Actors{{< /gloss >}} that this Worker hosts. What the event costs depends on what each Actor was doing when the node went away. From ad41886cf1007f73fbb998156d680563e3c3e7e3 Mon Sep 17 00:00:00 2001 From: Rachael Graham Date: Mon, 12 Oct 2026 16:29:18 -0500 Subject: [PATCH 2/2] Clarify the session quiescence recovery section Name the leader-election difference outright: quiescence claims run on every controller replica, unlike the reconciliation that "Run multiple controller replicas" describes, so the two sections no longer read as a contradiction. Rename the heading to "Recover missed Session pauses" and keep the explicit anchor, lead with what the pause is for before the commit and claim mechanics, and drop "workers" for the four concurrent claims, which collided with Agent Substrate Workers on this same page. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Rachael Graham --- .../1.x/operations/operational-considerations.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs-site/content/kagent/1.x/operations/operational-considerations.md b/docs-site/content/kagent/1.x/operations/operational-considerations.md index ecc43322..c049d8f7 100644 --- a/docs-site/content/kagent/1.x/operations/operational-considerations.md +++ b/docs-site/content/kagent/1.x/operations/operational-considerations.md @@ -97,13 +97,15 @@ Leader election keeps the replicas from conflicting. One replica holds a Kuberne > [!IMPORTANT] > If an installation runs on the development database, extra controller replicas do not make the installation highly available. That database is a single pod backed by one PersistentVolumeClaim, so losing it stops every replica at once. Run an external PostgreSQL instance before counting controller replicas toward availability. -## Tune session quiescence recovery {#session-quiescence-recovery} +## Recover missed Session pauses {#session-quiescence-recovery} -When a task settles, the controller commits the task history to PostgreSQL. Then the controller wakes local workers that pause or suspend the Session's {{< gloss "Actor" >}}Actor{{< /gloss >}}. Committed settlements do not wait for the recovery scan interval. Four workers drain durable claims, and PostgreSQL coordinates the work across controller replicas so that one replica handles each claim. +An idle Session's {{< gloss "Actor" >}}Actor{{< /gloss >}} is paused or suspended so that the capacity it holds returns to its pool. The controller starts this within moments of a turn finishing. The setting in this section governs only how quickly the controller recovers when it misses that moment. -If a controller exits after the commit and before the local wake-up, another replica finds the durable claim on its next recovery scan. The default recovery interval is `1m`. Increasing the interval can delay pause, suspension, and {{< gloss "Snapshot" >}}snapshot{{< /gloss >}} availability after a missed wake-up. With a shorter interval, the controller scans for missed work more often. +When a task settles, the controller commits the task history to PostgreSQL. It then signals itself to pause or suspend the Session's Actor. The signal is local to one controller process, so a controller that exits between the commit and the signal leaves the work undone. Another replica finds the record on its next recovery scan, which runs every `1m` by default. -Use `KAGENT_SESSION_QUIESCENCE_POLL_INTERVAL` to change only the missed-wake-up recovery interval. The value must be a positive Go duration. The controller retries temporarily blocked work and database errors within one second, regardless of the recovery interval. +This work is not leader-only, unlike the reconciliation that [Run multiple controller replicas](#run-multiple-controller-replicas) describes. Every replica claims from the same PostgreSQL queue, and PostgreSQL grants each record to one claimant, so no two replicas pause the same Session. Each replica runs four claims at a time, which bounds how much of this work one replica takes on. + +Set `KAGENT_SESSION_QUIESCENCE_POLL_INTERVAL` to change the recovery scan interval. A longer interval delays a pause, a suspend, and the {{< gloss "Snapshot" >}}snapshot{{< /gloss >}} that follows one, but only after a missed signal. A shorter interval scans more often. The value must be a positive Go duration, and the controller retries blocked work and database errors within one second regardless of the interval. ```yaml controller: