Skip to content

fix: propagate provider finish errors to headless runs - #113

Merged
byapparov merged 3 commits into
mainfrom
backport/issue-108
Sep 14, 2026
Merged

byapparov merged 3 commits into
mainfrom
backport/issue-108

Conversation

@byapparov

Copy link
Copy Markdown
Contributor

Closes #108

Intent

A provider can finish a streamed turn with a normalized error or content-filter reason without throwing an exception. Previously that path could emit no session failure and exit headless mode with status 0.

Expected Impact on Users

Headless consumers receive a non-zero exit and structured session error for unsuccessful provider turns. Successful empty stop turns and existing cancellation/idle-timeout behavior remain distinct.

Expected Outcomes

  • Provider error finishes persist as assistant errors and publish the session error lifecycle.
  • Partial tool activity and usage remain available for diagnosis.
  • Terminal events and process status agree without duplicate completion events.

Implementation

  • Map normalized error and content-filter finishes into the existing non-retryable API error lifecycle, retaining the finish reason in metadata.
  • Keep the terminal-reason matrix and versioned event documentation aligned with the implementation.

Scope Caveat

This does not add automatic retries or capture raw provider response diagnostics; those are separate proposals in #109 and #111.

Test Plan

  • Deterministic streamed error/content-filter fixtures cover headless non-zero exit and session failure emission.
  • Empty successful stop, tool activity, usage, cancellation, and idle-timeout regressions remain covered.

Verification

  • 28 focused and affected lifecycle tests passed.
  • CLI typecheck, formatting, and diff checks passed.

Risks and Rollout

The change uses the existing error/event contract and has no migration or feature flag. Roll back the commit if a provider-specific finish reason is found to be incorrectly classified.

@byapparov byapparov added this to the Enterprise Observability milestone Sep 14, 2026
Comment thread packages/cli/src/cli/cmd/run.errors.ts Outdated
}
if (status && status >= 500 && status < 600) {
return { reason: "provider", code: String(status), message }
if (name === "APIError" || (status && status >= 500 && status < 600)) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 APIError catch-all misclassifies 401/429 as "provider".

🤖 Fix with your agent
Fix this code review finding (aictrl-dev/cli PR #113, packages/cli/src/cli/cmd/run.errors.ts:33-34):

Problem: APIError catch-all misclassifies 401/429 as "provider"
Detail: The new `name === "APIError"` catch-all is status-blind and sits after only the message-regex branches. A thrown APIError whose status lives in data.statusCode but whose message misses the English regexes (Google's 429 "Resource has been exhausted", Anthropic's "rate_limit_error" — no space, 403 "Permission denied") now classifies as reason "provider" instead of "unknown". The new tests only use regex-friendly messages ("Rate limit exceeded", "Invalid API key"), so any reason-keyed retry/backoff, exit-code, or user-hint behavior treats real quota/auth APIErrors as generic provider failures, and data.isRetryable is ignored entirely.
Suggested fix: Before the APIError catch-all, classify by extracted status: 429 (and 408) → rate_limit/timeout, 401/403 → auth; or read err.data.statusCode and data.isRetryable inside the APIError branch instead of relying on message regexes. Add test cases with non-regex messages ("Resource has been exhausted", "rate_limit_error").

Implement the fix on the PR head branch and add a regression test that fails before the fix and passes after.
Why this matters

The new name === "APIError" catch-all is status-blind and sits after only the message-regex branches. A thrown APIError whose status lives in data.statusCode but whose message misses the English regexes (Google's 429 "Resource has been exhausted", Anthropic's "rate_limit_error" — no space, 403 "Permission denied") now classifies as reason "provider" instead of "unknown". The new tests only use regex-friendly messages ("Rate limit exceeded", "Invalid API key"), so any reason-keyed retry/backoff, exit-code, or user-hint behavior treats real quota/auth APIErrors as generic provider failures, and data.isRetryable is ignored entirely.

  if (/heap out of memory|ENOMEM/i.test(message)) {
    return { reason: "oom", message }
  }
  if (name === "APIError" || (status && status >= 500 && status < 600)) {
    return { reason: "provider", code: status ? String(status) : undefined, message }
  }
  return { reason: "unknown", code: status ? String(status) : undefined, message }
}

Comment thread EVENTS.md Outdated

Errors are attributed to the originating session. A child error alone does not change the primary session's exit status if the primary agent handles it successfully.

Release regression coverage: from `packages/cli`, run `bun test test/cli/run-provider-finish.test.ts test/cli/run-signal-cancellation.test.ts test/cli/classify-session-error.test.ts`. The provider fixture uses real Gemini SSE responses and the pinned SDK in a headless subprocess, including malformed function calls, content filtering, empty success, tool calls, partial output, and output limits.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Maintainer test instructions embedded in schema doc.

Suggested change
Release regression coverage: from `packages/cli`, run `bun test test/cli/run-provider-finish.test.ts test/cli/run-signal-cancellation.test.ts test/cli/classify-session-error.test.ts`. The provider fixture uses real Gemini SSE responses and the pinned SDK in a headless subprocess, including malformed function calls, content filtering, empty success, tool calls, partial output, and output limits.
Drop the "Release regression coverage" paragraph from EVENTS.md (keep it in the PR description or a maintainer testing doc), and consider moving the "Terminal reason semantics" table below the full message_complete field list so the `tokens` field definition stays adjacent to `usageStatus`/`finish`.
🤖 Fix with your agent
Fix this code review finding (aictrl-dev/cli PR #113, EVENTS.md:229):

Problem: Maintainer test instructions embedded in schema doc
Detail: EVENTS.md is a schema reference for event consumers, but the new section ends with a "Release regression coverage" paragraph instructing maintainers to run `bun test ...`. That is PR-checklist content, references a signal-cancellation test irrelevant to finish-reason semantics, and will rot as test files are renamed. It also splits the message_complete field enumeration, separating the `tokens` field doc from `usageStatus`/`finish` by ~20 lines.
Suggested fix: Drop the "Release regression coverage" paragraph from EVENTS.md (keep it in the PR description or a maintainer testing doc), and consider moving the "Terminal reason semantics" table below the full message_complete field list so the `tokens` field definition stays adjacent to `usageStatus`/`finish`.

Implement the fix on the PR head branch and add a regression test that fails before the fix and passes after.
Why this matters

EVENTS.md is a schema reference for event consumers, but the new section ends with a "Release regression coverage" paragraph instructing maintainers to run bun test .... That is PR-checklist content, references a signal-cancellation test irrelevant to finish-reason semantics, and will rot as test files are renamed. It also splits the message_complete field enumeration, separating the tokens field doc from usageStatus/finish by ~20 lines.


Errors are attributed to the originating session. A child error alone does not change the primary session's exit status if the primary agent handles it successfully.

Release regression coverage: from `packages/cli`, run `bun test test/cli/run-provider-finish.test.ts test/cli/run-signal-cancellation.test.ts test/cli/classify-session-error.test.ts`. The provider fixture uses real Gemini SSE responses and the pinned SDK in a headless subprocess, including malformed function calls, content filtering, empty success, tool calls, partial output, and output limits.

**`tokens`** (5-way breakdown, mirrors upstream `LLM.Usage`):

cost: usage.cost,
})
await Session.updateMessage(input.assistantMessage)
if (input.assistantMessage.error) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Error break may kill interactive sessions.

🤖 Fix with your agent
Fix this code review finding (aictrl-dev/cli PR #113, packages/cli/src/session/processor.ts:284-290):

Problem: Error break may kill interactive sessions
Detail: On a failed finish reason the code publishes Session.Event.Error and unconditionally breaks the loop. In interactive (non-headless) sessions a content-filter turn previously ended the turn and returned the prompt; now the session loop terminates, and a TUI listener for Session.Event.Error may additionally surface the error already persisted on the message (double handling). The thrown-error path goes through the catch block instead, so the two failure lifecycles this change claims to unify may still diverge (the catch path may recover/continue in interactive mode). Only the headless path is tested.
Suggested fix: Verify the interactive Session.Event.Error consumer and the catch block's recovery behavior; if interactive sessions should survive, gate the break on headless/invocation mode (or route through the same catch/failure handler a thrown nonretryable error uses). Add an interactive-mode test.

Implement the fix on the PR head branch and add a regression test that fails before the fix and passes after.
Why this matters

On a failed finish reason the code publishes Session.Event.Error and unconditionally breaks the loop. In interactive (non-headless) sessions a content-filter turn previously ended the turn and returned the prompt; now the session loop terminates, and a TUI listener for Session.Event.Error may additionally surface the error already persisted on the message (double handling). The thrown-error path goes through the catch block instead, so the two failure lifecycles this change claims to unify may still diverge (the catch path may recover/continue in interactive mode). Only the headless path is tested.

                    cost: usage.cost,
                  })
                  await Session.updateMessage(input.assistantMessage)
                  if (input.assistantMessage.error) {
                    await Bus.publish(Session.Event.Error, {
                      sessionID: input.sessionID,
                      error: input.assistantMessage.error,
                    })
                    break
                  }
                  if (snapshot) {

// Providers can end a successful HTTP stream with a failed
// model turn. Preserve its parts and usage, but use the same
// failure lifecycle as a thrown, nonretryable provider error.
if (value.finishReason === "error" || value.finishReason === "content-filter") {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Finish reasons other than error/content-filter still exit 0.

🤖 Fix with your agent
Fix this code review finding (aictrl-dev/cli PR #113, packages/cli/src/session/processor.ts:258):

Problem: Finish reasons other than error/content-filter still exit 0
Detail: Only "error" and "content-filter" are treated as failures. Gemini emits other failure finishReasons (RECITATION, BLOCKLIST, SPII, OTHER) that may normalize to "other"/"unknown"; those turns are still persisted as success and headless runs exit 0, so #108-style silent failures remain for a subset of blocked/malformed responses. The tests cover only MALFORMED_FUNCTION_CALL and SAFETY mappings.
Suggested fix: Verify how the provider SDK normalizes RECITATION/BLOCKLIST/OTHER; either map them into the failure set, or explicitly document the supported failure reasons in EVENTS.md and log unmapped ones. Extend tests with a RECITATION/OTHER case to pin the intended behavior.

Implement the fix on the PR head branch and add a regression test that fails before the fix and passes after.
Why this matters

Only "error" and "content-filter" are treated as failures. Gemini emits other failure finishReasons (RECITATION, BLOCKLIST, SPII, OTHER) that may normalize to "other"/"unknown"; those turns are still persisted as success and headless runs exit 0, so #108-style silent failures remain for a subset of blocked/malformed responses. The tests cover only MALFORMED_FUNCTION_CALL and SAFETY mappings.

                  // Providers can end a successful HTTP stream with a failed
                  // model turn. Preserve its parts and usage, but use the same
                  // failure lifecycle as a thrown, nonretryable provider error.
                  if (value.finishReason === "error" || value.finishReason === "content-filter") {
                    log.error("provider finish", {
                      sessionID: input.sessionID,
                      messageID: input.assistantMessage.id,
                      finishReason: value.finishReason,
                    })
                    input.assistantMessage.error = new MessageV2.APIError({
                      message:
                        value.finishReason === "content-filter"
                          ? "The provider blocked the response with a content filter."
                          : "The provider ended the response with an error finish reason.",
                      isRetryable: false,
                      metadata: { finishReason: value.finishReason },
                    }).toObject()
                  }

@aictrl-dev

aictrl-dev Bot commented Sep 14, 2026

Copy link
Copy Markdown

Code review

Verdict: Address the major findings before merging. · 🔴 0 · 🟠 1 · 🟡 3 · ⚪ 0 · 0/4 resolved

  • 🟡 EVENTS.md:229 — Maintainer test instructions embedded in schema doc
  • 🟠 packages/cli/src/cli/cmd/run.errors.ts:33-34 — APIError catch-all misclassifies 401/429 as "provider"
  • 🟡 packages/cli/src/session/processor.ts:258 — Finish reasons other than error/content-filter still exit 0
  • 🟡 packages/cli/src/session/processor.ts:284-290 — Error break may kill interactive sessions
🤖 Fix all 4 open findings with your agent
Fix the following code review findings on aictrl-dev/cli PR #113 (head branch).
Run the relevant tests/linters after each change.

1. EVENTS.md:229 — Maintainer test instructions embedded in schema doc
   Detail: EVENTS.md is a schema reference for event consumers, but the new section ends with a "Release regression coverage" paragraph instructing maintainers to run `bun test ...`. That is PR-checklist content, references a signal-cancellation test irrelevant to finish-reason semantics, and will rot as test files are renamed. It also splits the message_complete field enumeration, separating the `tokens` field doc from `usageStatus`/`finish` by ~20 lines.
   Suggested fix: Drop the "Release regression coverage" paragraph from EVENTS.md (keep it in the PR description or a maintainer testing doc), and consider moving the "Terminal reason semantics" table below the full message_complete field list so the `tokens` field definition stays adjacent to `usageStatus`/`finish`.
2. packages/cli/src/cli/cmd/run.errors.ts:33-34 — APIError catch-all misclassifies 401/429 as "provider"
   Detail: The new `name === "APIError"` catch-all is status-blind and sits after only the message-regex branches. A thrown APIError whose status lives in data.statusCode but whose message misses the English regexes (Google's 429 "Resource has been exhausted", Anthropic's "rate_limit_error" — no space, 403 "Permission denied") now classifies as reason "provider" instead of "unknown". The new tests only use regex-friendly messages ("Rate limit exceeded", "Invalid API key"), so any reason-keyed retry/backoff, exit-code, or user-hint behavior treats real quota/auth APIErrors as generic provider failures, and data.isRetryable is ignored entirely.
   Suggested fix: Before the APIError catch-all, classify by extracted status: 429 (and 408) → rate_limit/timeout, 401/403 → auth; or read err.data.statusCode and data.isRetryable inside the APIError branch instead of relying on message regexes. Add test cases with non-regex messages ("Resource has been exhausted", "rate_limit_error").
3. packages/cli/src/session/processor.ts:258 — Finish reasons other than error/content-filter still exit 0
   Detail: Only "error" and "content-filter" are treated as failures. Gemini emits other failure finishReasons (RECITATION, BLOCKLIST, SPII, OTHER) that may normalize to "other"/"unknown"; those turns are still persisted as success and headless runs exit 0, so #108-style silent failures remain for a subset of blocked/malformed responses. The tests cover only MALFORMED_FUNCTION_CALL and SAFETY mappings.
   Suggested fix: Verify how the provider SDK normalizes RECITATION/BLOCKLIST/OTHER; either map them into the failure set, or explicitly document the supported failure reasons in EVENTS.md and log unmapped ones. Extend tests with a RECITATION/OTHER case to pin the intended behavior.
4. packages/cli/src/session/processor.ts:284-290 — Error break may kill interactive sessions
   Detail: On a failed finish reason the code publishes Session.Event.Error and unconditionally breaks the loop. In interactive (non-headless) sessions a content-filter turn previously ended the turn and returned the prompt; now the session loop terminates, and a TUI listener for Session.Event.Error may additionally surface the error already persisted on the message (double handling). The thrown-error path goes through the catch block instead, so the two failure lifecycles this change claims to unify may still diverge (the catch path may recover/continue in interactive mode). Only the headless path is tested.
   Suggested fix: Verify the interactive Session.Event.Error consumer and the catch block's recovery behavior; if interactive sessions should survive, gate the break on headless/invocation mode (or route through the same catch/failure handler a thrown nonretryable error uses). Add an interactive-mode test.
📋 Out-of-diff findings (4)
Sev Location Finding
🟡 EVENTS.md:229 Maintainer test instructions embedded in schema doc
🟠 packages/cli/src/cli/cmd/run.errors.ts:33-34 APIError catch-all misclassifies 401/429 as "provider"
🟡 packages/cli/src/session/processor.ts:258 Finish reasons other than error/content-filter still exit 0
🟡 packages/cli/src/session/processor.ts:284-290 Error break may kill interactive sessions

Reviewed 6 files · 0 inline · view all 4 findings ↗


aictrl · AI code review for fast-moving teams · aictrl.dev

@byapparov

Copy link
Copy Markdown
Contributor Author

Review response — PR #113

Verified all four structured findings against the PR head; clarified documentation and added exact provider regressions without changing runtime behavior.

Issues addressed (pushed to this PR)

  • Maintainer instructions interrupt the event schema reference — moved release checks to CONTRIBUTING.md and placed terminal semantics after the complete message_complete field reference (commit bc7f75fe9b).

Review claims verified false (no change needed)

  • "APIError catch-all misclassifies 401/429"extractStatus already reads data.statusCode; the status 429 and 401/403 branches run before the APIError fallback. Added the exact Resource has been exhausted, rate_limit_error, and Permission denied examples: they pass without runtime changes. Retry eligibility belongs to SessionRetry, not this event-reason classifier.
  • "Error break may kill interactive sessions" — this headless repository has no TUI event consumer. The switch break reaches the shared error guard, cleanup, and stop return. The existing nonretryable catch follows the same path; SessionPrompt restores idle and retains the session for subsequent prompts. No process or session deletion is introduced here.

Not addressed here

  • Other finish reasonsOTHER retains the explicitly documented existing behavior in Fix successful exit on provider error finish reasons #108; expanding classification remains deferred pending a provider policy. The listed RECITATION, BLOCKLIST, and SPII examples already map to content-filter in pinned Google SDK 2.0.54 and fail. Added real-SSE subprocess cases to prove those mappings and the retained OTHER behavior.

Validation: 32 focused tests passed, CLI typecheck passed, Prettier and diff checks passed. MCP finding persistence is unavailable: matched 0, written 0, verified 0, failed 0, unrecorded 4. The structured verdicts are retained in this comment.

Comment thread packages/cli/src/cli/cmd/run.errors.ts Outdated
}
if (status && status >= 500 && status < 600) {
return { reason: "provider", code: String(status), message }
if (name === "APIError" || (status && status >= 500 && status < 600)) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 APIError catch-all folds statusful 4xx into "provider".

🤖 Fix with your agent
Fix this code review finding (aictrl-dev/cli PR #113, packages/cli/src/cli/cmd/run.errors.ts:33-35):

Problem: APIError catch-all folds statusful 4xx into "provider"
Detail: Adding `name === "APIError"` to the provider branch makes every APIError classify as reason "provider", including status-bearing non-5xx HTTP failures (400 invalid request, 402, 404) that previously fell through to reason "unknown". 401/403/429 are still classified correctly by the earlier status checks, but the catch-all is broader than the PR's stated need (classifying status-less persisted finish-reason errors) and changes session_error.reason telemetry semantics for pre-existing thrown 4xx provider errors consumed by CI dashboards.
Suggested fix: Tighten the condition so only status-less APIErrors (the persisted finish-reason errors) get the provider catch-all, e.g. `if ((name === "APIError" && !status) || (status && status >= 500 && status < 600))`, or key on `data?.metadata?.finishReason` instead of the error name.

Implement the fix on the PR head branch and add a regression test that fails before the fix and passes after.
Why this matters

Adding name === "APIError" to the provider branch makes every APIError classify as reason "provider", including status-bearing non-5xx HTTP failures (400 invalid request, 402, 404) that previously fell through to reason "unknown". 401/403/429 are still classified correctly by the earlier status checks, but the catch-all is broader than the PR's stated need (classifying status-less persisted finish-reason errors) and changes session_error.reason telemetry semantics for pre-existing thrown 4xx provider errors consumed by CI dashboards.

  if (name === \"APIError\" || (status && status >= 500 && status < 600)) {\n    return { reason: \"provider\", code: status ? String(status) : undefined, message }\n  }\n  return { reason: \"unknown\", code: status ? String(status) : undefined, message }\n}

// model turn. Preserve its parts and usage, but use the same
// failure lifecycle as a thrown, nonretryable provider error.
if (value.finishReason === "error" || value.finishReason === "content-filter") {
log.error("provider finish", {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 "other" finish abnormal terminations still exit 0.

Suggested change
log.error("provider finish", {
Verify the pinned @ai-sdk/google maps all Gemini failure finishReasons (PROHIBITED_CONTENT, LANGUAGE, FINISH_REASON_UNSPECIFIED) to "content-filter"/"error", and consider emitting a non-fatal warning event (or failing when the turn has no output) for "other" finishes so CI can distinguish abnormal termination from STOP.
🤖 Fix with your agent
Fix this code review finding (aictrl-dev/cli PR #113, packages/cli/src/session/processor.ts:259-273):

Problem: "other" finish abnormal terminations still exit 0
Detail: Only "error" and "content-filter" convert to a failed turn; a stream that terminates abnormally with a normalized "other" (Gemini OTHER, and unverified Gemini safety reasons like PROHIBITED_CONTENT/LANGUAGE depending on SDK mapping) still persists status "completed" and headless runs exit 0 — the same silent-CI-failure class issue #108 targets. The new test enshrines OTHER → exit 0, and EVENTS.md documents "other nonempty finish" as non-failure, so this is a deliberate policy — but the residual gap means some blocked/malformed responses remain indistinguishable from success in CI. Worth either verifying every Gemini failure finishReason maps to content-filter/error in the pinned SDK, or surfacing "other" terminations via a non-fatal warning event / failure when the turn produced no output.
Suggested fix: Verify the pinned @ai-sdk/google maps all Gemini failure finishReasons (PROHIBITED_CONTENT, LANGUAGE, FINISH_REASON_UNSPECIFIED) to "content-filter"/"error", and consider emitting a non-fatal warning event (or failing when the turn has no output) for "other" finishes so CI can distinguish abnormal termination from STOP.

Implement the fix on the PR head branch and add a regression test that fails before the fix and passes after.
Why this matters

Only "error" and "content-filter" convert to a failed turn; a stream that terminates abnormally with a normalized "other" (Gemini OTHER, and unverified Gemini safety reasons like PROHIBITED_CONTENT/LANGUAGE depending on SDK mapping) still persists status "completed" and headless runs exit 0 — the same silent-CI-failure class issue #108 targets. The new test enshrines OTHER → exit 0, and EVENTS.md documents "other nonempty finish" as non-failure, so this is a deliberate policy — but the residual gap means some blocked/malformed responses remain indistinguishable from success in CI. Worth either verifying every Gemini failure finishReason maps to content-filter/error in the pinned SDK, or surfacing "other" terminations via a non-fatal warning event / failure when the turn produced no output.

                  // Providers can end a successful HTTP stream with a failed\n                  // model turn. Preserve its parts and usage, but use the same\n                  // failure lifecycle as a thrown, nonretryable provider error.\n                  if (value.finishReason === "error" || value.finishReason === "content-filter") {\n                    log.error("provider finish", {\n                      sessionID: input.sessionID,\n                      messageID: input.assistantMessage.id,\n                      finishReason: value.finishReason,\n                    })

await Session.updateMessage(input.assistantMessage)
if (input.assistantMessage.error) {
await Bus.publish(Session.Event.Error, {
sessionID: input.sessionID,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Session.Event.Error publish awaited inconsistently.

Suggested change
sessionID: input.sessionID,
Await Bus.publish(Session.Event.Error, ...) at the max-retry branch (~L396) and the terminal catch branch (~L413) to match the new finish-step pattern, or extract a small awaited publishSessionError() helper used by all three sites.
🤖 Fix with your agent
Fix this code review finding (aictrl-dev/cli PR #113, packages/cli/src/session/processor.ts:286-289):

Problem: Session.Event.Error publish awaited inconsistently
Detail: The new finish-step failure path awaits Bus.publish(Session.Event.Error) (processor.ts:286), but the two sibling sites publishing the identical event in the same function remain fire-and-forget: the max-retry branch (~L396) and the terminal catch branch (~L413) call Bus.publish without await. Since this PR's stated goal is consistent failure events and headless consumers depend on session_error flushing before idle/exit, the same event should be published the same way at all three sites; un-awaited publishes rely on incidental later awaits to flush.
Suggested fix: Await Bus.publish(Session.Event.Error, ...) at the max-retry branch (~L396) and the terminal catch branch (~L413) to match the new finish-step pattern, or extract a small awaited publishSessionError() helper used by all three sites.

Implement the fix on the PR head branch and add a regression test that fails before the fix and passes after.
Why this matters

The new finish-step failure path awaits Bus.publish(Session.Event.Error) (processor.ts:286), but the two sibling sites publishing the identical event in the same function remain fire-and-forget: the max-retry branch (~L396) and the terminal catch branch (~L413) call Bus.publish without await. Since this PR's stated goal is consistent failure events and headless consumers depend on session_error flushing before idle/exit, the same event should be published the same way at all three sites; un-awaited publishes rely on incidental later awaits to flush.

                  await Session.updateMessage(input.assistantMessage)\n                  if (input.assistantMessage.error) {\n                    await Bus.publish(Session.Event.Error, {\n                      sessionID: input.sessionID,\n                      error: input.assistantMessage.error,\n                    })\n                    break\n                  }\n                  if (snapshot) {

[401, "Unauthenticated", "auth"],
[403, "Permission denied", "auth"],
[undefined, "Stream timeout", "timeout"],
] as const)("APIError preserves specific classifications (%s, %s)", (statusCode, message, reason) => {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test.each title omits expected reason placeholder.

--- a/packages/cli/test/cli/classify-session-error.test.ts
+++ b/packages/cli/test/cli/classify-session-error.test.ts
@@ -70,7 +70,7 @@
     [403, "Permission denied", "auth"],
     [undefined, "Stream timeout", "timeout"],
-  ] as const)("APIError preserves specific classifications (%s, %s)", (statusCode, message, reason) => {
+  ] as const)("APIError preserves specific classifications (%s, %s) → %s", (statusCode, message, reason) => {
     const res = classifySessionError({ name: "APIError", data: { statusCode, message, isRetryable: false } })
     expect(res.reason).toBe(reason)
🤖 Fix with your agent
Fix this code review finding (aictrl-dev/cli PR #113, packages/cli/test/cli/classify-session-error.test.ts:73):

Problem: test.each title omits expected reason placeholder
Detail: The test.each title "APIError preserves specific classifications (%s, %s)" interpolates only statusCode and message; the third parameter — the expected reason, the most important value — never appears in generated test names or failure output (e.g. a failure renders as "(undefined, Stream timeout)" without "timeout").
Suggested fix: Use "APIError preserves specific classifications (%s, %s) → %s" so the expected reason appears in each generated test name.

Suggested patch:
--- a/packages/cli/test/cli/classify-session-error.test.ts
+++ b/packages/cli/test/cli/classify-session-error.test.ts
@@ -70,7 +70,7 @@
     [403, "Permission denied", "auth"],
     [undefined, "Stream timeout", "timeout"],
-  ] as const)("APIError preserves specific classifications (%s, %s)", (statusCode, message, reason) => {
+  ] as const)("APIError preserves specific classifications (%s, %s) → %s", (statusCode, message, reason) => {
     const res = classifySessionError({ name: "APIError", data: { statusCode, message, isRetryable: false } })
     expect(res.reason).toBe(reason)


Implement the fix on the PR head branch and add a regression test that fails before the fix and passes after.
Why this matters

The test.each title "APIError preserves specific classifications (%s, %s)" interpolates only statusCode and message; the third parameter — the expected reason, the most important value — never appears in generated test names or failure output (e.g. a failure renders as "(undefined, Stream timeout)" without "timeout").

  test.each([\n    [undefined, \"The provider ended the response with an error finish reason.\", \"provider\"],\n    [429, \"Rate limit exceeded\", \"rate_limit\"],\n    [429, \"Resource has been exhausted\", \"rate_limit\"],\n    [429, \"rate_limit_error\", \"rate_limit\"],\n    [401, \"Invalid API key\", \"auth\"],\n    [401, \"Unauthenticated\", \"auth\"],\n    [403, \"Permission denied\", \"auth\"],\n    [undefined, \"Stream timeout\", \"timeout\"],\n  ] as const)(\"APIError preserves specific classifications (%s, %s)\", (statusCode, message, reason) => {

content: {
role: "model",
parts: [
{ functionCall: { name: "read", args: { filePath: path.join(tmp.path, "aictrl.json") } } },

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fetch handler references tmp before declaration.

Suggested change
{ functionCall: { name: "read", args: { filePath: path.join(tmp.path, "aictrl.json") } } },
Move the `await using tmp = await tmpdir({...})` setup above the Bun.serve call so the fetch handler only references already-initialized bindings.
🤖 Fix with your agent
Fix this code review finding (aictrl-dev/cli PR #113, packages/cli/test/cli/run-provider-finish.test.ts:37):

Problem: fetch handler references tmp before declaration
Detail: The Bun.serve fetch handler references `tmp.path` inside the tool-call branch, but `await using tmp = await tmpdir(...)` is declared ~34 lines later, after the server is already listening. It only works because no request can arrive before tmp initializes (random port, CLI spawned afterwards), but the closure-over-later-const is a fragile use-before-declare pattern — any request arriving in the setup window would throw a TDZ ReferenceError inside the handler.
Suggested fix: Move the `await using tmp = await tmpdir({...})` setup above the Bun.serve call so the fetch handler only references already-initialized bindings.

Implement the fix on the PR head branch and add a regression test that fails before the fix and passes after.
Why this matters

The Bun.serve fetch handler references tmp.path inside the tool-call branch, but await using tmp = await tmpdir(...) is declared ~34 lines later, after the server is already listening. It only works because no request can arrive before tmp initializes (random port, CLI spawned afterwards), but the closure-over-later-const is a fragile use-before-declare pattern — any request arriving in the setup window would throw a TDZ ReferenceError inside the handler.

      const server = Bun.serve({\n        port: 0,\n        fetch(): Response {\n          calls++\n          const chunks =\n            tool && calls === 1\n              ? [\n                  {\n                    candidates: [\n                      {\n                        index: 0,\n                        content: {\n                          role: \"model\",\n                          parts: [\n                            { functionCall: { name: \"read\", args: { filePath: path.join(tmp.path, \"aictrl.json\") } } },

@aictrl-dev

aictrl-dev Bot commented Sep 14, 2026

Copy link
Copy Markdown

Code review

Verdict: Looks good — only minor / nit comments below. · 🔴 0 · 🟠 0 · 🟡 3 · ⚪ 2 · 0/5 resolved

  • 🟡 packages/cli/src/cli/cmd/run.errors.ts:33-35 — APIError catch-all folds statusful 4xx into "provider"
  • 🟡 packages/cli/src/session/processor.ts:259-273 — "other" finish abnormal terminations still exit 0
  • 🟡 packages/cli/src/session/processor.ts:286-289 — Session.Event.Error publish awaited inconsistently
  • packages/cli/test/cli/classify-session-error.test.ts:73 — test.each title omits expected reason placeholder
  • packages/cli/test/cli/run-provider-finish.test.ts:37 — fetch handler references tmp before declaration
🤖 Fix all 5 open findings with your agent
Fix the following code review findings on aictrl-dev/cli PR #113 (head branch).
Run the relevant tests/linters after each change.

1. packages/cli/src/cli/cmd/run.errors.ts:33-35 — APIError catch-all folds statusful 4xx into "provider"
   Detail: Adding `name === "APIError"` to the provider branch makes every APIError classify as reason "provider", including status-bearing non-5xx HTTP failures (400 invalid request, 402, 404) that previously fell through to reason "unknown". 401/403/429 are still classified correctly by the earlier status checks, but the catch-all is broader than the PR's stated need (classifying status-less persisted finish-reason errors) and changes session_error.reason telemetry semantics for pre-existing thrown 4xx provider errors consumed by CI dashboards.
   Suggested fix: Tighten the condition so only status-less APIErrors (the persisted finish-reason errors) get the provider catch-all, e.g. `if ((name === "APIError" && !status) || (status && status >= 500 && status < 600))`, or key on `data?.metadata?.finishReason` instead of the error name.
2. packages/cli/src/session/processor.ts:259-273 — "other" finish abnormal terminations still exit 0
   Detail: Only "error" and "content-filter" convert to a failed turn; a stream that terminates abnormally with a normalized "other" (Gemini OTHER, and unverified Gemini safety reasons like PROHIBITED_CONTENT/LANGUAGE depending on SDK mapping) still persists status "completed" and headless runs exit 0 — the same silent-CI-failure class issue #108 targets. The new test enshrines OTHER → exit 0, and EVENTS.md documents "other nonempty finish" as non-failure, so this is a deliberate policy — but the residual gap means some blocked/malformed responses remain indistinguishable from success in CI. Worth either verifying every Gemini failure finishReason maps to content-filter/error in the pinned SDK, or surfacing "other" terminations via a non-fatal warning event / failure when the turn produced no output.
   Suggested fix: Verify the pinned @ai-sdk/google maps all Gemini failure finishReasons (PROHIBITED_CONTENT, LANGUAGE, FINISH_REASON_UNSPECIFIED) to "content-filter"/"error", and consider emitting a non-fatal warning event (or failing when the turn has no output) for "other" finishes so CI can distinguish abnormal termination from STOP.
3. packages/cli/src/session/processor.ts:286-289 — Session.Event.Error publish awaited inconsistently
   Detail: The new finish-step failure path awaits Bus.publish(Session.Event.Error) (processor.ts:286), but the two sibling sites publishing the identical event in the same function remain fire-and-forget: the max-retry branch (~L396) and the terminal catch branch (~L413) call Bus.publish without await. Since this PR's stated goal is consistent failure events and headless consumers depend on session_error flushing before idle/exit, the same event should be published the same way at all three sites; un-awaited publishes rely on incidental later awaits to flush.
   Suggested fix: Await Bus.publish(Session.Event.Error, ...) at the max-retry branch (~L396) and the terminal catch branch (~L413) to match the new finish-step pattern, or extract a small awaited publishSessionError() helper used by all three sites.
4. packages/cli/test/cli/classify-session-error.test.ts:73 — test.each title omits expected reason placeholder
   Detail: The test.each title "APIError preserves specific classifications (%s, %s)" interpolates only statusCode and message; the third parameter — the expected reason, the most important value — never appears in generated test names or failure output (e.g. a failure renders as "(undefined, Stream timeout)" without "timeout").
   Suggested fix: Use "APIError preserves specific classifications (%s, %s) → %s" so the expected reason appears in each generated test name.
5. packages/cli/test/cli/run-provider-finish.test.ts:37 — fetch handler references tmp before declaration
   Detail: The Bun.serve fetch handler references `tmp.path` inside the tool-call branch, but `await using tmp = await tmpdir(...)` is declared ~34 lines later, after the server is already listening. It only works because no request can arrive before tmp initializes (random port, CLI spawned afterwards), but the closure-over-later-const is a fragile use-before-declare pattern — any request arriving in the setup window would throw a TDZ ReferenceError inside the handler.
   Suggested fix: Move the `await using tmp = await tmpdir({...})` setup above the Bun.serve call so the fetch handler only references already-initialized bindings.
📋 Out-of-diff findings (5)
Sev Location Finding
🟡 packages/cli/src/cli/cmd/run.errors.ts:33-35 APIError catch-all folds statusful 4xx into "provider"
🟡 packages/cli/src/session/processor.ts:259-273 "other" finish abnormal terminations still exit 0
🟡 packages/cli/src/session/processor.ts:286-289 Session.Event.Error publish awaited inconsistently
packages/cli/test/cli/classify-session-error.test.ts:73 test.each title omits expected reason placeholder
packages/cli/test/cli/run-provider-finish.test.ts:37 fetch handler references tmp before declaration

Reviewed 7 files · 0 inline · view all 5 findings ↗


aictrl · AI code review for fast-moving teams · aictrl.dev

@byapparov byapparov self-assigned this Sep 14, 2026
@byapparov byapparov added the bug Something isn't working label Sep 14, 2026
@byapparov

Copy link
Copy Markdown
Contributor Author

Review response — PR #113

Verified the five findings from the latest review and pushed the scoped fixes in 00f0cd30da.

Issues addressed (pushed to this PR)

  • Status-bearing 4xx classification — the APIError fallback now applies only when status is absent. Added 400/402/404 regressions preserving unknown; 401/403/429 and status-less finish failures keep their existing categories.
  • Inconsistent error publication — both the max-retry and terminal-catch paths now await subscribers. Two blocking-subscriber tests verify completion ordering.
  • Classification test titles — generated names now include the expected reason.
  • Fixture initialization order — create the temporary directory before starting the server, then write configuration using the allocated port.

All four fixes are in commit 00f0cd30da.

Review claims verified false (no change needed)

None in this review round.

Not addressed here

  • Broader other/unknown failure policy — retained the documented Fix successful exit on provider error finish reasons #108 boundary. Verified the actual pinned Google adapter with 12 finish-reason fixtures: PROHIBITED_CONTENT maps to content-filter, LANGUAGE maps to unknown, and FINISH_REASON_UNSPECIFIED/OTHER map to other. Added end-to-end subprocess coverage for PROHIBITED_CONTENT and FINISH_REASON_UNSPECIFIED, plus explicit event documentation. message_complete.finish already distinguishes other from stop; adding a warning event or failing empty output would require a separate contract decision.

Validation: 71 tests passed, CLI typecheck passed, Prettier and diff checks passed. Running the new classification and publication regressions against the previous runtime produced 5 expected failures; the fixes pass all five.

Persistence is unavailable without the MCP connection: matched 0, written 0, verified 0, failed 0, unrecorded 5. The verdicts are preserved below.

@github-actions

Copy link
Copy Markdown

Review

Verified the fix end-to-end: read the processor/run/classification changes, traced the headless event lifecycle, ran the full packages/cli suite (161 CLI + 140 session tests, 0 fail), typecheck, and independently confirmed the pinned @ai-sdk/google@2.0.54 finish-reason mappings against node_modules (they match EVENTS.md exactly, including LANGUAGEunknown via the default branch).

Correctness — confirmed:

  • Lifecycle ordering is sound: the error is persisted on the assistant message (processor.ts:283) before Session.Event.Error is published (processor.ts:285), and the terminal message (time.completed + final updateMessage) is persisted before the prompt loop's deferred idle, so session_errormessage_complete(status: "error")session_complete.error reach JSON consumers in order with exit 1. The terminal() guard makes completion idempotent, so no duplicate terminal events even when retries and signals interleave.
  • await Bus.publish (processor.ts:396, processor.ts:413) fixes a real fire-and-forget race where the processor could persist terminal state and return before in-process subscribers finished. Bus.publish catches per-subscriber errors, so a failing subscriber can't reject the processor, and the headless subscription path goes through synchronous GlobalBus.emit, so there's no deadlock risk from awaiting.
  • Classification precedence in classifySessionError is right: 429/401/403/ProviderAuthError/timeout/oom checks run before the new status-less APIErrorprovider rule, so specific classifications win (covered by the new table tests).
  • The json_schema path can't clobber the provider error — the !processor.message.error guard at prompt.ts:713 prevents overwriting with StructuredOutputError.
  • The error path still records the snapshot patch and marks pending tool parts as errored via the post-loop code (processor.ts:418-448), so partial output/tools/usage survive as documented.
  • Unreachable-combination check: needsCompaction + error can't co-occur (the error path breaks before the overflow check, and a prior compaction would have already returned "compact"), so the if (needsCompaction) return "compact" ordering at processor.ts:453 is safe.
  • Cancellation/timeout remain distinct: an abort still surfaces as MessageAbortedError (message_complete.status: "aborted"), never reclassified as a provider finish — regression tests cover this.

Minor, non-blocking:

  1. processor.ts:289 — the break inside the switch is redundant; the loop-exit actually happens at processor.ts:377 (if (needsCompaction || input.assistantMessage.error) break). If that line-377 check were ever refactored away, the switch break alone would not stop stream consumption. Worth a comment tying them together.
  2. The finish-error path skips SessionSummary.summarize and the compaction-overflow check (processor.ts:305-313). Skipping compaction is fine since you return "stop", but failed turns also get no summary — fine if intended, just noting it's implicit.
  3. classifySessionError now attributes all status-less APIErrors to provider, not just finish-reason errors — e.g. parseStreamError results and a final non-retried ECONNRESET (message-v2.ts:831-843) shift unknownprovider. This is consistent with the provider-fault intent, just flagging the blast radius is wider than the changelog line implies.
  4. An APIError whose message matches /timeout/i classifies as timeout rather than provider (regex check runs first) — the tests document this as intended, but worth knowing since Gemini stream timeouts wrapped in APIError would land there.

No bugs, security issues, or reliability regressions found. LGTM.

Reviewed SHA: 00f0cd3

@byapparov
byapparov merged commit 7924f58 into main Sep 14, 2026
5 checks passed
@byapparov
byapparov deleted the backport/issue-108 branch September 14, 2026 16:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix successful exit on provider error finish reasons

1 participant