Skip to content

fix(session): allow explicit deletion after task settlement failure - #3213

Open
Jacob953 wants to merge 6 commits into
kagent-dev:mainfrom
Jacob953:fix/session-unpublished-boundary
Open

Jacob953 wants to merge 6 commits into
kagent-dev:mainfrom
Jacob953:fix/session-unpublished-boundary

Conversation

@Jacob953

@Jacob953 Jacob953 commented Oct 15, 2026 •

Copy link
Copy Markdown

When a runtime saves a terminal or waiting task boundary but its SettleTask acknowledgement never commits, the Session rejects further work and lifecycle operations, including deletion. This PR allows a caller to delete that Session through the existing durable Actor deletion workflow.

Related issue: #3010, case B: unsettled task boundary blocks Delete. This implements the issue's explicit-deletion alternative. It partially addresses #3010; automatic publication and recovery of the original Session remain open.

Ready for maintainer review of the deletion semantics and retained lifecycle guards.

Implementation

  • Share one runtime admission query across deletion, dispatch, task writes and checkpoint creation. Only explicit DELETE may proceed despite unpublished task events. Live dispatch reservations, claimed quiescence work and checkpoint creation still block deletion. The existing workflow records deletion intent before backend cleanup and retains it for retry if cleanup fails.
  • Serialize deletion admission and task settlement under the same Session row lock. If deletion starts first, late settlement of an unpublished boundary returns ErrConflict (Aborted over gRPC), including after deletion preparation fails. It cannot publish task state/history or enqueue push notifications. A previously published acknowledgement remains idempotent until final deletion.
  • Keep idle expiration blocked by unpublished events. The explicit-deletion exception therefore does not allow TTL cleanup to discard an unconfirmed boundary.
  • Document the behavior and remaining lifecycle guards in architecture documentation and source comments. The change is confined to Session lifecycle admission and settlement; it adds no API or schema.

Deletion discards the pending result and removes the Session's runtime through the existing deletion workflow. It does not resume the conversation. Orphaned active tasks and abandoned quiescence claims require separate recovery work.

Validation

Coverage Result and method
Database and Session workflow PostgreSQL-backed tests cover all four terminal states, INPUT_REQUIRED and AUTH_REQUIRED, both initial creation and later updates, deletion/settlement ordering, idempotence, callback suppression and workflow retries after partial failure. Passed under the race detector.
Go ADK settlement failure The production Go ADK executor runs through A2A and TaskStore gRPC with PostgreSQL. Four injected settlement failures exhaust retries and leave a private completed task/public working task. Deletion admission succeeds and late settlement is rejected. The model is an in-process fixture. Passed.
Live Actor deletion The committed TestSessionUnpublishedBoundaryDeletion stages completed and input-required boundaries through TaskStore while a real Actor is blocked at a mock model request. It checks Actor removal, rejected stale writes, a lost successful public deletion reply and no compute recreation. Go ADK cases passed on Kind 1.37.0, Substrate 0.5.0-alpha2 and gVisor on arm64.
Existing suites Full database, migration, Session service and gRPC race suites passed without test exclusions. Inline SQL preparation and make -C go lint passed.

The TTL regression test also now takes its precise cutoffs from PostgreSQL, matching the event clock. This removes a host/container clock-skew failure in the test without changing production TTL timing. It passed 100 consecutive race runs.

Relevant commands, from go/ with Docker and the repository's envtest assets configured:

go test -race -count=1 ./core/internal/database ./core/pkg/migrations
go test -race -count=1 ./core/internal/service/session ./core/internal/grpcserver
go test -race -count=100 ./core/internal/database -run '^TestSessionExpirationRechecksActivity$'

With the clean-install E2E fixtures configured as described in go/core/test/e2e/README.md:

go test -race -count=1 -timeout 15m ./core/test/e2e -run '^TestSessionUnpublishedBoundaryDeletion$/^kagent$'

An additional local infrastructure probe verified that deletion intent survives backend unavailability and controller replacement, and a fresh request completes deletion after recovery. That probe is not a committed regression test. Lost Substrate deletion responses are covered by workflow fault tests; the live E2E test injects loss of the public API reply after success.

Pending review and validation

  • Maintainer agreement on allowing explicit deletion to supersede an unpublished boundary, with the remaining guards retained.
  • Upstream CI requires maintainer approval. The shared E2E test includes Codex, Claude and BYO cases, but only Go ADK was run live locally. The BYO fixture uses Go ADK through the BYO compiler. The live Kind results predate the shared-query refactor; the current head still needs the CI harness-matrix run.

AI assistance: OpenAI Codex assisted with implementation, testing and this PR description.

Keep explicit deletion available after a runtime loses its settlement callback. Preserve dispatch, checkpoint and claimed lifecycle exclusions. Verify failed deletion and a lost response after deletion with PostgreSQL-backed workflow tests.

Refs kagent-dev#3010. This restores deletion, not in-place conversation recovery.

AI assistance: implementation and validation assisted by OpenAI Codex.

Signed-off-by: Junfeng Yu <64075563+Jacob953@users.noreply.github.com>
Reproduce exhausted Go ADK settlement retries through gRPC and PostgreSQL. Preserve deletion write exclusions and document the separate recovery gap for abandoned quiescence claims. Link the shared tracking issues and state the conditions required before relaxing recovery guards.

AI assistance: implementation and validation assisted by OpenAI Codex.

Signed-off-by: Junfeng Yu <64075563+Jacob953@users.noreply.github.com>
Preserve idle expiration cleanup checks while allowing the explicit deletion escape for kagent-dev#3010 B. Reject late publication after deletion admission, including preparation failure, and retain idempotent published acknowledgements. Cover all boundary states, initial and updated boundaries, failure retries and settlement/deletion ordering.

Automatic publication and same-session recovery remain outside this deletion fix. AI assistance: implementation and validation assisted by OpenAI Codex.

Signed-off-by: Junfeng Yu <64075563+Jacob953@users.noreply.github.com>
@github-actions github-actions Bot added the bug Something isn't working label Oct 15, 2026
Use PostgreSQL clock cutoffs in the TTL regression to avoid host/container skew. Add shared E2E coverage for deleting staged terminal and waiting boundaries while a real Actor is running, including a lost public reply and rejected stale writes. Document the harness selectors.

Validated locally with the Go ADK harness on Kind/Substrate/gVisor, full affected race suites, 100 TTL repetitions, and Go lint. AI assistance: implementation and validation assisted by OpenAI Codex.

Signed-off-by: Junfeng Yu <64075563+Jacob953@users.noreply.github.com>
@Jacob953
Jacob953 marked this pull request as ready for review October 15, 2026 06:29
@Jacob953
Jacob953 requested a review from a team as a code owner October 15, 2026 06:29
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 15, 2026
@Jacob953
Jacob953 marked this pull request as draft October 15, 2026 07:02
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 15, 2026
@Jacob953
Jacob953 marked this pull request as ready for review October 15, 2026 07:19
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 15, 2026
Comment thread go/core/internal/database/session_operations.go Outdated
Comment thread go/core/internal/database/session_operations.go Outdated
Use one admission query for lifecycle operations, dispatch, task writes and
checkpoint creation. Allow unpublished boundaries only for explicit deletion,
while retaining claimed quiescence and live dispatch protections.

Remove fork tracking and unrelated cancellation references from source comments.
Verify explicit deletion still rejects live dispatch and checkpoint creation.

Refs kagent-dev#3010.

AI assistance: implementation and validation assisted by OpenAI Codex.

Signed-off-by: Junfeng Yu <64075563+Jacob953@users.noreply.github.com>
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 16, 2026
Wait for the executor cleanup callback instead of counting telemetry flushes.
Join the test A2A server and check its exit error. Assert the precise public
task state after settlement retries fail.

Clarify the deletion fault fixture's effect ordering and verify its lifecycle
interface at compile time. Tighten comments and document the limit on
settlement acknowledgement idempotence after deletion.

Refs kagent-dev#3010.

AI assistance: implementation and validation assisted by OpenAI Codex.

Signed-off-by: Junfeng Yu <64075563+Jacob953@users.noreply.github.com>
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 16, 2026
@Jacob953
Jacob953 requested a review from moezdil October 19, 2026 02:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants