Skip to content

fix(engine): record harness tool names, failure kinds and Claude compactions - #3799

Merged
LHagfoss merged 4 commits into
mainfrom
fix/engine-harness-telemetry-gaps
Nov 15, 2026
Merged

LHagfoss merged 4 commits into
mainfrom
fix/engine-harness-telemetry-gaps

Conversation

@LHagfoss

Copy link
Copy Markdown
Owner

What

Analysis of the last three days of threads showed the data store missing or mislabeling much of what external harnesses do. This fixes how Claude Code and Codex turns are recorded.

  • Compactions were missed. The Claude Code ACP adapter now sends compaction facts under _meta.jetbrains.air.contextCompaction, in an update with no status or title. The detector only knew the older claudeCode.toolName = "compact" shape. The store held 2 compaction rows (both failed) while 37 manual /compact runs were captured. The detector now reads the new shape. Migration 13 backfills the missed rows from captured session/update payloads, once, without touching existing rows.
  • Tool names. tool_calls.tool_name is the ACP kind for external harnesses, so 757 MCP calls in three days were just other. New column harness_tool (migration 12) holds the harness's own name: Claude's Bash, Read, mcp__rustcode__wait_session; Codex's item type or mcp__<server>__<tool>, which the Codex bridge now sets as _meta.rustcode.toolName. mcp_server is filled from it.
  • Tool failure reasons. Failed ACP tool calls had no error_kind. They now get exit_status, permission_denied, guard_refused, timeout, interrupted (cut off by the turn end) or failed. Claude's Exit code N result text now fills exit_code.
  • Turn failure reasons. New turns.error_kind (migration 12) classifies a failed external turn's error as unknown_session, rate_limited, usage_limit, auth, harness_exited, network, and so on. The error text itself is never stored.
  • command_name skips a leading cd dir && (about a third of shell calls were recorded as cd).
  • Docs: docs/data-store.md and the query_data description now explain that Codex input_tokens already include the cached tokens and Codex reports no cost or cache writes, so cross-harness comparisons need adjusting.

Changelog

Usage data now records Claude Code compactions (recovering past ones), each tool's real name, and why a tool call or turn failed.

Testing

  • The workspace type check (--workspace --tests) and cargo fmt --all -- --check pass locally.
  • New unit tests cover the compaction shape, the backfill migration (run twice, one row), tool names, failure kinds, exit-code parsing and command_name. The tests themselves run in CI.
  • The backfill SELECT was run read-only against the live store and returned the 37 compactions.

@github-actions github-actions Bot added the fix A bug fix label Nov 15, 2026
@LHagfoss
LHagfoss merged commit 07ad218 into main Nov 15, 2026
6 checks passed
@LHagfoss
LHagfoss deleted the fix/engine-harness-telemetry-gaps branch November 15, 2026 13:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant