Skip to content

fix: preserve headless tool-call failure truth - #116

Open
byapparov wants to merge 2 commits into
mainfrom
backport/headless-truth
Open

byapparov wants to merge 2 commits into
mainfrom
backport/headless-truth

Conversation

@byapparov

Copy link
Copy Markdown
Contributor

Relates to #89

Intent

A model can return a terminal-looking stop after a local tool call, and a child task can fail while its parent only sees empty text. Those cases can terminate headless execution with incomplete truth.

Expected Impact on Users

Headless runs continue after ordinary local tool calls when structured output is still pending, and failed child tasks are surfaced to the parent instead of being presented as successful empty output.

Expected Outcomes

  • Tool calls that were not provider-executed prevent premature terminal exit.
  • Structured-output turns continue through local tool execution and validate the subsequent model response.
  • Child assistant and tool failures retain task identity and propagate as actionable errors.

Implementation

  • Persist provider-executed attribution on tool parts and use it in prompt continuation guards.
  • Preserve child failure details in TaskTool results while retaining the existing success output shape.

Scope Caveat

This prepares the malformed-tool-call recovery work in #111; it does not ship automatic retries or claim live recovery effectiveness.

Test Plan

  • Local tool-call and structured-output subprocess fixtures cover continuation after a stop reason.
  • Child assistant/tool failure tests cover attribution and the existing successful task format.

Verification

  • 41 focused tests passed.
  • CLI typecheck, formatting, and diff checks passed.

Risks and Rollout

The change is limited to headless continuation and child-task error propagation. No event schema removal or retry policy change is involved.

@byapparov byapparov added this to the Enterprise Observability milestone Sep 14, 2026

export function taskResultText(result: MessageV2.WithParts, sessionID: string) {
if (result.info.role === "assistant" && result.info.error) {
const data = result.info.error.data

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 taskResultText crashes when error.data is undefined.

🤖 Fix with your agent
Fix this code review finding (aictrl-dev/cli PR #116, packages/cli/src/tool/task.ts:30-31):

Problem: taskResultText crashes when error.data is undefined
Detail: taskResultText reads `const data = result.info.error.data` and then applies `"message" in data` without guarding against `data` being undefined or a non-object. Persisted child errors whose serialized shape lacks a `data` field (or carries a non-object data) make the `in` operator throw `TypeError: Cannot use 'in' operator`, replacing the intended "Subagent failed (task_id: ...): <reason>" message with a confusing TypeError — undermining this PR's goal of surfacing the real child failure cause. The added test only covers `MessageV2.APIError.toObject()`, which happens to include `data`, so the gap is untested. Repro: given a subagent whose final assistant message persists an error without a `data` payload, when the parent runs the Task tool, then taskResultText throws the raw TypeError instead of the child's actual error name/message.
Suggested fix: Guard the operand before using `in`: `const data = result.info.error.data as { message?: string } | undefined` then `const message = data && typeof data.message === "string" ? data.message : result.info.error.name` (or `"message" in (data ?? {})`). Add a test for an error serialized without `data`.

Implement the fix on the PR head branch and add a regression test that fails before the fix and passes after.
Why this matters

taskResultText reads const data = result.info.error.data and then applies "message" in data without guarding against data being undefined or a non-object. Persisted child errors whose serialized shape lacks a data field (or carries a non-object data) make the in operator throw TypeError: Cannot use 'in' operator, replacing the intended "Subagent failed (task_id: ...): " message with a confusing TypeError — undermining this PR's goal of surfacing the real child failure cause. The added test only covers MessageV2.APIError.toObject(), which happens to include data, so the gap is untested. Repro: given a subagent whose final assistant message persists an error without a data payload, when the parent runs the Task tool, then taskResultText throws the raw TypeError instead of the child's actual error name/message.

export function taskResultText(result: MessageV2.WithParts, sessionID: string) {
  if (result.info.role === "assistant" && result.info.error) {
    const data = result.info.error.data
    const message = "message" in data && typeof data.message === "string" ? data.message : result.info.error.name
    throw new Error(`Subagent failed (task_id: ${sessionID}): ${message}`)
  }
  const failed = result.parts.findLast((part) => part.type === "tool" && part.state.status === "error")

},
},
metadata: value.providerMetadata,
metadata: {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 providerExecuted magic string lacks shared constant.

🤖 Fix with your agent
Fix this code review finding (aictrl-dev/cli PR #116, packages/cli/src/session/processor.ts:148-150):

Problem: providerExecuted magic string lacks shared constant
Detail: The `providerExecuted` key is written into tool-part metadata in processor.ts and read back with a magic string in prompt.ts (`part.metadata?.providerExecuted`, packages/cli/src/session/prompt.ts:66), creating an implicit cross-file contract with no shared constant or typed field. The new tests also hardcode the string. A typo or rename on either side would silently disable the headless-truth logic with no compiler error.
Suggested fix: Export a shared constant (e.g. `MessageV2.PROVIDER_EXECUTED_METADATA_KEY = "providerExecuted"`) or add a typed optional field on ToolPart, and use it in both processor.ts (write), prompt.ts (read), and the tests.

Implement the fix on the PR head branch and add a regression test that fails before the fix and passes after.
Why this matters

The providerExecuted key is written into tool-part metadata in processor.ts and read back with a magic string in prompt.ts (part.metadata?.providerExecuted, packages/cli/src/session/prompt.ts:66), creating an implicit cross-file contract with no shared constant or typed field. The new tests also hardcode the string. A typo or rename on either side would silently disable the headless-truth logic with no compiler error.

                           start: Date.now(),
                         },
                       },
                      metadata: {
                        ...value.providerMetadata,
                        ...(value.providerExecuted ? { providerExecuted: true } : {}),
                      },
                      })
                      toolcalls[value.toolCallId] = part as MessageV2.ToolPart

},
},
metadata: value.providerMetadata,
metadata: {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 providerMetadata can forge providerExecuted flag.

🤖 Fix with your agent
Fix this code review finding (aictrl-dev/cli PR #116, packages/cli/src/session/processor.ts:148-151):

Problem: providerMetadata can forge providerExecuted flag
Detail: When `value.providerExecuted` is falsy, the persisted metadata is just `{ ...value.providerMetadata }`; a provider-supplied top-level key `providerExecuted` inside providerMetadata survives verbatim. SessionPrompt.hasToolCalls (packages/cli/src/session/prompt.ts:66) then treats the tool call as provider-executed, so the prompt loop can exit without executing the tool or taking the follow-up model turn — silently dropping a local tool call and, in json_schema mode, raising StructuredOutputError instead of continuing. Repro: given a provider adapter that surfaces a top-level `providerExecuted: true` entry in a tool-call part's providerMetadata while the part's own providerExecuted flag is false, when the model streams that local tool call and finishes with reason "stop", then the CLI persists metadata.providerExecuted=true, hasToolCalls returns false, and the loop exits without executing the tool.
Suggested fix: Overwrite the key unconditionally instead of conditionally spreading, e.g. `metadata: { ...value.providerMetadata, providerExecuted: value.providerExecuted === true }` (or strip any incoming `providerExecuted` key from providerMetadata before merging), so the flag can only come from the stream part itself.

Implement the fix on the PR head branch and add a regression test that fails before the fix and passes after.
Why this matters

When value.providerExecuted is falsy, the persisted metadata is just { ...value.providerMetadata }; a provider-supplied top-level key providerExecuted inside providerMetadata survives verbatim. SessionPrompt.hasToolCalls (packages/cli/src/session/prompt.ts:66) then treats the tool call as provider-executed, so the prompt loop can exit without executing the tool or taking the follow-up model turn — silently dropping a local tool call and, in json_schema mode, raising StructuredOutputError instead of continuing. Repro: given a provider adapter that surfaces a top-level providerExecuted: true entry in a tool-call part's providerMetadata while the part's own providerExecuted flag is false, when the model streams that local tool call and finishes with reason "stop", then the CLI persists metadata.providerExecuted=true, hasToolCalls returns false, and the loop exits without executing the tool.

                           start: Date.now(),
                         },
                       },
                      metadata: {
                        ...value.providerMetadata,
                        ...(value.providerExecuted ? { providerExecuted: true } : {}),
                      },
                      })
                      toolcalls[value.toolCallId] = part as MessageV2.ToolPart


// Check if model finished (finish reason is not "tool-calls" or "unknown")
const modelFinished = processor.message.finish && !["tool-calls", "unknown"].includes(processor.message.finish)
const hasCurrentToolCalls = hasToolCalls(await MessageV2.parts(processor.message.id))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Parts query runs even when model not finished.

🤖 Fix with your agent
Fix this code review finding (aictrl-dev/cli PR #116, packages/cli/src/session/prompt.ts:718-720):

Problem: Parts query runs even when model not finished
Detail: hasCurrentToolCalls awaits `MessageV2.parts(processor.message.id)` unconditionally on every iteration of the structured-output retry loop, even when `modelFinished` is false or `processor.message.error` is set and the value is never used. This is inconsistent with the short-circuit `&&` chain it feeds and adds a redundant async persistence read per retry turn.
Suggested fix: Short-circuit inside the condition so the await only runs when the other guards pass: `if (modelFinished && !processor.message.error && !hasToolCalls(await MessageV2.parts(processor.message.id))) { ... }`, or compute hasCurrentToolCalls inside an `if (modelFinished)` guard.

Implement the fix on the PR head branch and add a regression test that fails before the fix and passes after.
Why this matters

hasCurrentToolCalls awaits MessageV2.parts(processor.message.id) unconditionally on every iteration of the structured-output retry loop, even when modelFinished is false or processor.message.error is set and the value is never used. This is inconsistent with the short-circuit && chain it feeds and adds a redundant async persistence read per retry turn.

      // Check if model finished (finish reason is not "tool-calls" or "unknown")
      const modelFinished = processor.message.finish && !["tool-calls", "unknown"].includes(processor.message.finish)
      const hasCurrentToolCalls = hasToolCalls(await MessageV2.parts(processor.message.id))

      if (modelFinished && !hasCurrentToolCalls && !processor.message.error) {

@@ -150,7 +163,7 @@ export const TaskTool = Tool.define("task", async (ctx) => {
parts: promptParts,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 agent.subtask.complete skipped on child failure.

Suggested change
parts: promptParts,
Trigger `agent.subtask.complete` (with a success/error flag) before throwing, or wrap the taskResultText call so the plugin event fires in a finally block on the failure path.
🤖 Fix with your agent
Fix this code review finding (aictrl-dev/cli PR #116, packages/cli/src/tool/task.ts:163-165):

Problem: agent.subtask.complete skipped on child failure
Detail: taskResultText now throws on child failure before `await Plugin.trigger("agent.subtask.complete", ...)` executes, so plugins subscribed to subtask completion never observe failed subtasks, and any post-trigger accounting on this path is skipped. Previously the event fired for every completed subtask regardless of outcome; now failed subtasks become invisible to plugin-based stats/notifications. Repro: given a plugin subscribed to "agent.subtask.complete", when a child subagent fails (message-level error or error tool part), then taskResultText throws before Plugin.trigger runs and the plugin never receives the event.
Suggested fix: Trigger `agent.subtask.complete` (with a success/error flag) before throwing, or wrap the taskResultText call so the plugin event fires in a finally block on the failure path.

Implement the fix on the PR head branch and add a regression test that fails before the fix and passes after.
Why this matters

taskResultText now throws on child failure before await Plugin.trigger("agent.subtask.complete", ...) executes, so plugins subscribed to subtask completion never observe failed subtasks, and any post-trigger accounting on this path is skipped. Previously the event fired for every completed subtask regardless of outcome; now failed subtasks become invisible to plugin-based stats/notifications. Repro: given a plugin subscribed to "agent.subtask.complete", when a child subagent fails (message-level error or error tool part), then taskResultText throws before Plugin.trigger runs and the plugin never receives the event.

        parts: promptParts,
      })

      const text = taskResultText(result, session.id)

      await Plugin.trigger(
        "agent.subtask.complete",

@@ -709,8 +715,9 @@ export namespace SessionPrompt {

// Check if model finished (finish reason is not "tool-calls" or "unknown")
const modelFinished = processor.message.finish && !["tool-calls", "unknown"].includes(processor.message.finish)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finish-reason filter duplicated; extract helper.

Suggested change
const modelFinished = processor.message.finish && !["tool-calls", "unknown"].includes(processor.message.finish)
Add `export function isModelFinished(finish?: string) { return !!finish && !["tool-calls", "unknown"].includes(finish) }` next to hasToolCalls and use it in both the loop-exit condition and the modelFinished const.
🤖 Fix with your agent
Fix this code review finding (aictrl-dev/cli PR #116, packages/cli/src/session/prompt.ts:717):

Problem: Finish-reason filter duplicated; extract helper
Detail: This PR extracts hasToolCalls as a shared helper, but the adjacent finish-reason filter `!["tool-calls", "unknown"].includes(finish)` still appears twice in prompt.ts (the loop-exit condition around line 342 and the modelFinished computation at line 717). Extracting both predicates keeps the two exit paths symmetric and single-sourced; the two sites must stay in agreement for the loop logic to be correct.
Suggested fix: Add `export function isModelFinished(finish?: string) { return !!finish && !["tool-calls", "unknown"].includes(finish) }` next to hasToolCalls and use it in both the loop-exit condition and the modelFinished const.

Implement the fix on the PR head branch and add a regression test that fails before the fix and passes after.
Why this matters

This PR extracts hasToolCalls as a shared helper, but the adjacent finish-reason filter !["tool-calls", "unknown"].includes(finish) still appears twice in prompt.ts (the loop-exit condition around line 342 and the modelFinished computation at line 717). Extracting both predicates keeps the two exit paths symmetric and single-sourced; the two sites must stay in agreement for the loop logic to be correct.

      // Check if model finished (finish reason is not "tool-calls" or "unknown")
      const modelFinished = processor.message.finish && !["tool-calls", "unknown"].includes(processor.message.finish)
      const hasCurrentToolCalls = hasToolCalls(await MessageV2.parts(processor.message.id))

      if (modelFinished && !hasCurrentToolCalls && !processor.message.error) {

const message = "message" in data && typeof data.message === "string" ? data.message : result.info.error.name
throw new Error(`Subagent failed (task_id: ${sessionID}): ${message}`)
}
const failed = result.parts.findLast((part) => part.type === "tool" && part.state.status === "error")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicated tool error check; use type predicate.

🤖 Fix with your agent
Fix this code review finding (aictrl-dev/cli PR #116, packages/cli/src/tool/task.ts:34-35):

Problem: Duplicated tool error check; use type predicate
Detail: The findLast predicate and the immediately following if condition repeat the same compound check (`part.type === "tool" && part.state.status === "error"`) purely for TypeScript narrowing. A type predicate on findLast removes the duplicated condition.
Suggested fix: Use a type-guard predicate: `const failed = result.parts.findLast((part): part is MessageV2.ToolPart => part.type === "tool" && part.state.status === "error")` then `if (failed) { ... }`.

Implement the fix on the PR head branch and add a regression test that fails before the fix and passes after.
Why this matters

The findLast predicate and the immediately following if condition repeat the same compound check (part.type === "tool" && part.state.status === "error") purely for TypeScript narrowing. A type predicate on findLast removes the duplicated condition.

    throw new Error(`Subagent failed (task_id: ${sessionID}): ${message}`)
  }
  const failed = result.parts.findLast((part) => part.type === "tool" && part.state.status === "error")
  if (failed?.type === "tool" && failed.state.status === "error") {
    throw new Error(`Subagent failed (task_id: ${sessionID}): ${failed.state.error}`)
  }
  return result.parts.findLast((part) => part.type === "text")?.text ?? ""

@aictrl-dev

aictrl-dev Bot commented Sep 14, 2026

Copy link
Copy Markdown

Code review

Verdict: Address the major findings before merging. · 🔴 0 · 🟠 1 · 🟡 4 · ⚪ 2 · 0/7 resolved

  • 🟡 packages/cli/src/session/processor.ts:148-150 — providerExecuted magic string lacks shared constant
  • 🟡 packages/cli/src/session/processor.ts:148-151 — providerMetadata can forge providerExecuted flag
  • packages/cli/src/session/prompt.ts:717 — Finish-reason filter duplicated; extract helper
  • 🟡 packages/cli/src/session/prompt.ts:718-720 — Parts query runs even when model not finished
  • 🟠 packages/cli/src/tool/task.ts:30-31 — taskResultText crashes when error.data is undefined
  • packages/cli/src/tool/task.ts:34-35 — Duplicated tool error check; use type predicate
  • 🟡 packages/cli/src/tool/task.ts:163-165 — agent.subtask.complete skipped on child failure
🤖 Fix all 7 open findings with your agent
Fix the following code review findings on aictrl-dev/cli PR #116 (head branch).
Run the relevant tests/linters after each change.

1. packages/cli/src/session/processor.ts:148-150 — providerExecuted magic string lacks shared constant
   Detail: The `providerExecuted` key is written into tool-part metadata in processor.ts and read back with a magic string in prompt.ts (`part.metadata?.providerExecuted`, packages/cli/src/session/prompt.ts:66), creating an implicit cross-file contract with no shared constant or typed field. The new tests also hardcode the string. A typo or rename on either side would silently disable the headless-truth logic with no compiler error.
   Suggested fix: Export a shared constant (e.g. `MessageV2.PROVIDER_EXECUTED_METADATA_KEY = "providerExecuted"`) or add a typed optional field on ToolPart, and use it in both processor.ts (write), prompt.ts (read), and the tests.
2. packages/cli/src/session/processor.ts:148-151 — providerMetadata can forge providerExecuted flag
   Detail: When `value.providerExecuted` is falsy, the persisted metadata is just `{ ...value.providerMetadata }`; a provider-supplied top-level key `providerExecuted` inside providerMetadata survives verbatim. SessionPrompt.hasToolCalls (packages/cli/src/session/prompt.ts:66) then treats the tool call as provider-executed, so the prompt loop can exit without executing the tool or taking the follow-up model turn — silently dropping a local tool call and, in json_schema mode, raising StructuredOutputError instead of continuing. Repro: given a provider adapter that surfaces a top-level `providerExecuted: true` entry in a tool-call part's providerMetadata while the part's own providerExecuted flag is false, when the model streams that local tool call and finishes with reason "stop", then the CLI persists metadata.providerExecuted=true, hasToolCalls returns false, and the loop exits without executing the tool.
   Suggested fix: Overwrite the key unconditionally instead of conditionally spreading, e.g. `metadata: { ...value.providerMetadata, providerExecuted: value.providerExecuted === true }` (or strip any incoming `providerExecuted` key from providerMetadata before merging), so the flag can only come from the stream part itself.
3. packages/cli/src/session/prompt.ts:717 — Finish-reason filter duplicated; extract helper
   Detail: This PR extracts hasToolCalls as a shared helper, but the adjacent finish-reason filter `!["tool-calls", "unknown"].includes(finish)` still appears twice in prompt.ts (the loop-exit condition around line 342 and the modelFinished computation at line 717). Extracting both predicates keeps the two exit paths symmetric and single-sourced; the two sites must stay in agreement for the loop logic to be correct.
   Suggested fix: Add `export function isModelFinished(finish?: string) { return !!finish && !["tool-calls", "unknown"].includes(finish) }` next to hasToolCalls and use it in both the loop-exit condition and the modelFinished const.
4. packages/cli/src/session/prompt.ts:718-720 — Parts query runs even when model not finished
   Detail: hasCurrentToolCalls awaits `MessageV2.parts(processor.message.id)` unconditionally on every iteration of the structured-output retry loop, even when `modelFinished` is false or `processor.message.error` is set and the value is never used. This is inconsistent with the short-circuit `&&` chain it feeds and adds a redundant async persistence read per retry turn.
   Suggested fix: Short-circuit inside the condition so the await only runs when the other guards pass: `if (modelFinished && !processor.message.error && !hasToolCalls(await MessageV2.parts(processor.message.id))) { ... }`, or compute hasCurrentToolCalls inside an `if (modelFinished)` guard.
5. packages/cli/src/tool/task.ts:30-31 — taskResultText crashes when error.data is undefined
   Detail: taskResultText reads `const data = result.info.error.data` and then applies `"message" in data` without guarding against `data` being undefined or a non-object. Persisted child errors whose serialized shape lacks a `data` field (or carries a non-object data) make the `in` operator throw `TypeError: Cannot use 'in' operator`, replacing the intended "Subagent failed (task_id: ...): <reason>" message with a confusing TypeError — undermining this PR's goal of surfacing the real child failure cause. The added test only covers `MessageV2.APIError.toObject()`, which happens to include `data`, so the gap is untested. Repro: given a subagent whose final assistant message persists an error without a `data` payload, when the parent runs the Task tool, then taskResultText throws the raw TypeError instead of the child's actual error name/message.
   Suggested fix: Guard the operand before using `in`: `const data = result.info.error.data as { message?: string } | undefined` then `const message = data && typeof data.message === "string" ? data.message : result.info.error.name` (or `"message" in (data ?? {})`). Add a test for an error serialized without `data`.
6. packages/cli/src/tool/task.ts:34-35 — Duplicated tool error check; use type predicate
   Detail: The findLast predicate and the immediately following if condition repeat the same compound check (`part.type === "tool" && part.state.status === "error"`) purely for TypeScript narrowing. A type predicate on findLast removes the duplicated condition.
   Suggested fix: Use a type-guard predicate: `const failed = result.parts.findLast((part): part is MessageV2.ToolPart => part.type === "tool" && part.state.status === "error")` then `if (failed) { ... }`.
7. packages/cli/src/tool/task.ts:163-165 — agent.subtask.complete skipped on child failure
   Detail: taskResultText now throws on child failure before `await Plugin.trigger("agent.subtask.complete", ...)` executes, so plugins subscribed to subtask completion never observe failed subtasks, and any post-trigger accounting on this path is skipped. Previously the event fired for every completed subtask regardless of outcome; now failed subtasks become invisible to plugin-based stats/notifications. Repro: given a plugin subscribed to "agent.subtask.complete", when a child subagent fails (message-level error or error tool part), then taskResultText throws before Plugin.trigger runs and the plugin never receives the event.
   Suggested fix: Trigger `agent.subtask.complete` (with a success/error flag) before throwing, or wrap the taskResultText call so the plugin event fires in a finally block on the failure path.
📋 Out-of-diff findings (7)
Sev Location Finding
🟡 packages/cli/src/session/processor.ts:148-150 providerExecuted magic string lacks shared constant
🟡 packages/cli/src/session/processor.ts:148-151 providerMetadata can forge providerExecuted flag
packages/cli/src/session/prompt.ts:717 Finish-reason filter duplicated; extract helper
🟡 packages/cli/src/session/prompt.ts:718-720 Parts query runs even when model not finished
🟠 packages/cli/src/tool/task.ts:30-31 taskResultText crashes when error.data is undefined
packages/cli/src/tool/task.ts:34-35 Duplicated tool error check; use type predicate
🟡 packages/cli/src/tool/task.ts:163-165 agent.subtask.complete skipped on child failure

Reviewed 6 files · 0 inline · view all 7 findings ↗


aictrl · AI code review for fast-moving teams · aictrl.dev

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant