Evaluate bounded recovery from malformed model tool calls

[byapparov]

Context

An observed headless execution using CLI 0.4.3 and Gemini 2.5 Flash completed one file-read tool call, then emitted a terminal message with finish: "error". No final text was recorded; there was no session-error event, and the process exited 0. A downstream consumer classified the result as incomplete despite the successful process exit.

The pinned Google SDK 2.0.54 maps only MALFORMED_FUNCTION_CALL to normalized error. This is strong evidence for a malformed generated function call, but the raw response and exact invalid call were not retained. Do not describe the exact offending call or its cause as proven.

Problem / Goal

The CLI currently terminates this failure class, but there is no measured evidence that retrying is effective or what recovery is safe. Establish that evidence before making automatic retry a runtime default.

Proposed Approach

Compare fail-fast baseline, bounded regeneration, and explicit repair feedback for malformed function-call finishes. Separate a provider-rejected call that never executed from tool failures after side effects. Use deterministic fixtures for control-flow/safety guarantees and a small, cost-capped live evaluation to estimate model recovery. Record the chosen cap and corpus before running; synthetic results cannot establish live recovery effectiveness.

Acceptance Criteria

• Define the hypothesis, bounded evaluation corpus, request/cost cap, and candidate policies before live evaluation.
• Report recovery rate, added latency, known token/cost overhead, repeated-failure rate, and cases where the invalid call or diagnostic is unavailable, with sample counts and limitations.
• Evaluate Gemini 2.5 Flash for the incident class; do not claim to reproduce the exact production call without its missing payload.
• Prove with fixtures that already executed tool side effects are not replayed by the recovery strategy, including a turn with completed tools before a later error.
• Specify attempt/time/cost limits, cancellation behavior, and the final explicit failure outcome when limits are exhausted; permission denials, auth failures, and safety filtering are not silently retried as malformed calls.
• Produce a go/no-go recommendation with measurable acceptance thresholds and remaining unknowns. Open or scope implementation work only if the evidence supports it.
• Use the diagnostic and attempt telemetry contracts from the preceding tickets; a successful recovery preserves the failed-attempt history.

Out of Scope

Shipping a default automatic retry policy in this spike, promising a recovery SLO without data, switching all models, re-running the original execution, and broad tool-error recovery.

Roadmap Alignment

• Pillar: EXEC; Q3 2026 pilot-ready executor reliability and code-review quality.
• Priority: P2 — evaluate after truthful outcomes and diagnostics are available.
• Milestone: Enterprise Observability (CLI repository milestone; repository-specific milestone numbering).
• Uses existing repository labels; priority and pillar are recorded here because matching labels do not exist in this repository.

References

• Parent lifecycle tracker: Epic: Make headless NDJSON lifecycle and usage telemetry reliable #89; broader OTEL/export work: Comprehensive OpenTelemetry integration for observability #22.
• CLI v0.4.3: packages/cli/src/session/processor.ts, src/session/prompt.ts, src/cli/cmd/run.ts, and src/cli/cmd/run.invocation.ts (paths relative to packages/cli except the first).
• Pinned Google finish-reason mapping.
• Related earlier work: Fix: aictrl run exits 0 on auth failure, masking broken CI workflows #70 (exception-driven exit status), NDJSON event schema gaps for downstream observability consumers #63 (event contract), Add NDJSON invocation lifecycle events before session creation #90 (invocation lifecycle), Preserve unknown usage on failed headless turns #93 (unknown usage), Add model stream idle-timeout handling #80 (stream idle timeout).

Sequencing

Schedule after outcome, diagnostic, and attempt telemetry work: #108, #109, #110