diff --git a/docs/data-store.md b/docs/data-store.md index 24f949a76..44ba1e583 100644 --- a/docs/data-store.md +++ b/docs/data-store.md @@ -35,9 +35,9 @@ payload only when the user turns payload capture on ("Captured payloads"). | object | holds | | --- | --- | | `threads` | one row per thread: `id`, `title`, `project`, `cwd`, `harness`, `model`, `effort`, `parent_id`, `parent_title`, `created_at`, `updated_at`, and (migration 9) `account`, `machine`, `created_with_version`, `branch`, `final_commit`, `pull_request`, `pull_request_state`, `settled_at` | -| `turns` | one row per turn: `thread_id`, `started_at`, `finished_at`, `harness`, `model`, `input_tokens`, `output_tokens`, `cached_tokens`, `cache_write_tokens`, `cost_usd`, `tool_calls`, `status`, `duration_ms`, `time_to_first_token_ms`, `model_time_ms`, `tool_time_ms`, `context_tokens`, `context_window`, `stop_reason`, and (migration 9) `effort`, `account`, `machine`, `rustcode_version`, `diff_files`, `diff_added`, `diff_removed`, unique on `(thread_id, started_at)` | +| `turns` | one row per turn: `thread_id`, `started_at`, `finished_at`, `harness`, `model`, `input_tokens`, `output_tokens`, `cached_tokens`, `cache_write_tokens`, `cost_usd`, `tool_calls`, `status`, `duration_ms`, `time_to_first_token_ms`, `model_time_ms`, `tool_time_ms`, `context_tokens`, `context_window`, `stop_reason`, and (migration 9) `effort`, `account`, `machine`, `rustcode_version`, `diff_files`, `diff_added`, `diff_removed`, and (migration 12) `error_kind`, unique on `(thread_id, started_at)` | | `approvals` | one row per approval a user was asked for: `thread_id`, `source` (`builtin`, `subagent`, `harness`), `tool_name`, `requested_at`, `decided_at`, `wait_ms`, `decision` (`allow_once`, `allow_hour`, `allow_always`, `reject`, `cancelled`, or the option kind a harness offered) | -| `tool_calls` | one row per finished tool call: `thread_id`, `turn_id`, `tool_name`, `success`, `exit_code`, `at`, `started_at`, `duration_ms`, `output_bytes`, `truncated`, `mcp_server`, `command_name`, `error_kind` | +| `tool_calls` | one row per finished tool call: `thread_id`, `turn_id`, `tool_name`, `success`, `exit_code`, `at`, `started_at`, `duration_ms`, `output_bytes`, `truncated`, `mcp_server`, `command_name`, `error_kind`, and (migration 12) `harness_tool` | | `compactions` | one row per history compaction, from any harness: `thread_id`, `turn_id`, `harness`, `model`, `at`, `duration_ms`, `trigger`, `mode`, `tokens_before`, `tokens_after`, `messages_before`, `messages_after`, `idle_secs_before`, `count`, `success`, `error_kind`, `external_id`, `source` | | `daily_usage` | a view over `turns`: `day`, `harness`, `model`, summed tokens, `cost_usd` and turn count | | `daily_compactions` | a view over `compactions`: compactions, failures, tokens before and after, reclaimed tokens and mean duration per day, harness and trigger | @@ -131,6 +131,42 @@ FROM threads WHERE settled_at IS NOT NULL; SELECT decision, COUNT(*), AVG(wait_ms) FROM approvals GROUP BY 1; ``` +## Failures and tool names + +Migration 12 adds two columns, both `NULL` where nothing reported them: + +| column | meaning | +| --- | --- | +| `turns.error_kind` | why a failed external-harness turn failed, a short kind from its error: `unknown_session`, `rate_limited`, `usage_limit`, `auth`, `context_overflow`, `provider_error`, `timeout`, `harness_exited`, `harness_start`, `network`, `other`. The error text is not stored. The built-in harness keeps using `stop_reason` | +| `tool_calls.harness_tool` | an external harness's own name for the tool: Claude Code's `Bash`, `Read`, `mcp__rustcode__wait_session`; a Codex item type (`commandExecution`, `fileChange`) or `mcp____`. `tool_name` stays the ACP kind (`execute`, `other`), and `mcp_server` is filled from either | + +A failed external tool call's `error_kind` is `exit_status` (ran and exited +non-zero; Claude Code's `Exit code N` gives `exit_code` too), +`permission_denied` (the harness or its permission check refused it), +`guard_refused` (the command guard), `timeout`, `interrupted` (still running +when the turn ended) or `failed`. `command_name` skips a leading `cd dir &&` +and similar shell setup, so it names the program that did the work. + +Migration 13 recovers Claude Code compactions recorded before the detector +knew the adapter's `_meta.jetbrains.air.contextCompaction` shape, from the +captured `session/update` payloads that hold them. + +## Comparing harnesses + +Token columns are what each harness reported, and harnesses count +differently: + +- **Claude Code** reports `input_tokens` without the cache: `cached_tokens` + (cache reads) and `cache_write_tokens` are separate, and the three add up + to the prompt. +- **Codex** reports `input_tokens` with the cache read already inside it + (`cached_tokens` is the part of it that was cached), no cache writes, and + no cost: a ChatGPT sign-in is not billed per token. Compare a Codex + `input_tokens` with a Claude Code `input_tokens + cached_tokens + + cache_write_tokens`. +- A turn's `model` is the one it ran on, so a thread that switched models + has turns on each; group by `turns.model`, not `threads.model`. + ## Compactions Every compaction RustCode can see is one row, whichever harness ran it: @@ -139,7 +175,7 @@ Every compaction RustCode can see is one row, whichever harness ran it: | --- | --- | --- | | built-in harness | `AppState::record_compaction`, where the chat card and `context.compaction` event are written | `auto_context` (the history crossed its budget) or `manual` (`/compact`) | | Codex | the `contextCompaction` item (`item/completed`) | `manual` for a `/compact` turn, else `harness_auto` | -| Claude Code | the ACP adapter's synthetic `Compact conversation` tool call (`_meta.claudeCode.toolName = "compact"`), terminal update only | Claude's `trigger`: `manual`, `harness_auto` for its automatic pass, `cache_expiry` for the adapter's idle compaction | +| Claude Code | the ACP adapter's synthetic `Compact conversation` tool call: the facts update under `_meta.jetbrains.air.contextCompaction` (current adapter), or the terminal update with `_meta.claudeCode.toolName = "compact"` (older adapter) | Claude's `trigger`: `manual`, `harness_auto` for its automatic pass, `cache_expiry` for the adapter's idle compaction | `mode` is `ai_summary` or `deterministic_record` for a RustCode compaction and `harness_summary` when the harness made it. Token counts, message counts and diff --git a/rustcode/engine/src/command_guard.rs b/rustcode/engine/src/command_guard.rs index 7a96dc20c..b556af25c 100644 --- a/rustcode/engine/src/command_guard.rs +++ b/rustcode/engine/src/command_guard.rs @@ -695,13 +695,21 @@ pub fn shell_quote(word: &str) -> String { /// The first program in a shell command, peeling wrappers, assignments and shells. /// Only the program name is retained for tool-call telemetry. +/// A leading `cd dir &&` (or another builtin that only sets up the shell) is +/// skipped, so `cd repo && cargo check` records `cargo`; a command that is only +/// such builtins records the first of them. pub(crate) fn command_name(command: &str) -> Option { + const SETUP: &[&str] = &[ + "cd", "pushd", "popd", "export", "set", "source", ".", "true", ":", + ]; let mut found = Vec::new(); scan(command, None, 0, &Context::default(), &mut found); + let first = found.first().map(|invocation| invocation.program.clone()); found .into_iter() - .next() .map(|invocation| invocation.program) + .find(|program| !SETUP.contains(&program.as_str())) + .or(first) } /// The command an approval request asks to run, when it is a shell command: diff --git a/rustcode/engine/src/command_guard_tests.rs b/rustcode/engine/src/command_guard_tests.rs index 2e36de554..cc530a910 100644 --- a/rustcode/engine/src/command_guard_tests.rs +++ b/rustcode/engine/src/command_guard_tests.rs @@ -394,3 +394,14 @@ fn the_hook_denies_a_refused_bash_call_and_ignores_the_rest() { assert_eq!(hook_response(&call("Read", "git stash")), None); assert_eq!(hook_response("not json"), None); } + +#[test] +fn command_name_skips_a_leading_cd() { + assert_eq!( + command_name("cd /repo && cargo check").as_deref(), + Some("cargo") + ); + assert_eq!(command_name("cd /repo; git status").as_deref(), Some("git")); + assert_eq!(command_name("cd /repo").as_deref(), Some("cd")); + assert_eq!(command_name("rg -n foo").as_deref(), Some("rg")); +} diff --git a/rustcode/engine/src/data_store.rs b/rustcode/engine/src/data_store.rs index 7a23f5c2c..d90c81a26 100644 --- a/rustcode/engine/src/data_store.rs +++ b/rustcode/engine/src/data_store.rs @@ -367,6 +367,46 @@ const MIGRATIONS: &[(i64, &str, &str)] = &[ FROM tool_calls GROUP BY substr(at, 1, 10), tool_name, command_name;", ), + // Additive: why a turn failed, and an external harness's own tool name + // next to the ACP kind in `tool_name`. + ( + 12, + "failure_kinds_and_harness_tools", + "ALTER TABLE turns ADD COLUMN error_kind TEXT; + ALTER TABLE tool_calls ADD COLUMN harness_tool TEXT;", + ), + // Claude Code compactions the detector missed: the adapter moved their + // facts to `_meta.jetbrains.air.contextCompaction`. Where payload capture + // kept the update, it is the row; an existing row is left alone. + ( + 13, + "backfill_claude_compactions", + "INSERT INTO compactions (thread_id, harness, model, at, duration_ms, trigger, mode, + tokens_before, tokens_after, success, external_id, source) + SELECT m.thread_id, m.harness, + (SELECT model FROM threads WHERE id = m.thread_id), + m.at, + json_extract(m.payload_json, '$.params.update._meta.jetbrains.air.contextCompaction.durationMs'), + CASE json_extract(m.payload_json, '$.params.update._meta.jetbrains.air.contextCompaction.trigger') + WHEN 'manual' THEN 'manual' + WHEN 'idle' THEN 'cache_expiry' + ELSE 'harness_auto' END, + 'harness_summary', + json_extract(m.payload_json, '$.params.update._meta.jetbrains.air.contextCompaction.preTokens'), + json_extract(m.payload_json, '$.params.update._meta.jetbrains.air.contextCompaction.postTokens'), + 1, + json_extract(m.payload_json, '$.params.update.toolCallId'), + m.harness + FROM harness_rpc_messages m + WHERE m.thread_id IS NOT NULL + AND m.method = 'session/update' + AND m.payload_json LIKE '%contextCompaction%' + AND json_valid(m.payload_json) + AND json_extract(m.payload_json, '$.params.update.toolCallId') IS NOT NULL + AND (json_extract(m.payload_json, '$.params.update._meta.jetbrains.air.contextCompaction.trigger') IS NOT NULL + OR json_extract(m.payload_json, '$.params.update._meta.jetbrains.air.contextCompaction.preTokens') IS NOT NULL) + ON CONFLICT(thread_id, external_id) DO NOTHING;", + ), ]; /// One turn's token totals. Each is optional: a harness that reports no count @@ -500,6 +540,11 @@ pub struct ToolCallRow { pub truncated: Option, /// The MCP server in an `mcp____` name. pub mcp_server: Option, + /// An external harness's own name for the tool (`Bash`, `Read`, + /// `mcp__rustcode__wait_session`, a Codex item type), where `tool_name` + /// holds the ACP kind (`execute`, `other`). `None` for the built-in + /// harness, whose `tool_name` is already the tool's name. + pub harness_tool: Option, /// The first word of a shell command (`cargo`, `git`, `bun`), never the /// whole command. pub command_name: Option, @@ -545,6 +590,10 @@ pub struct TurnRow { pub diff_files: Option, pub diff_added: Option, pub diff_removed: Option, + /// Why a failed turn failed, as a short kind (`unknown_session`, + /// `harness_exited`, `rate_limited`, …; see [`turn_failure_kind`]), + /// never the error text. + pub error_kind: Option, pub tool_calls: Vec, } @@ -2143,9 +2192,9 @@ fn write_turn_in(transaction: &Connection, turn: &TurnRow) -> Result<(), String> cost_usd, tool_calls, status, duration_ms, time_to_first_token_ms, model_time_ms, tool_time_ms, context_tokens, context_window, stop_reason, effort, account, machine, rustcode_version, diff_files, diff_added, - diff_removed) + diff_removed, error_kind) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12, ?13, ?14, - ?15, ?16, ?17, ?18, ?19, ?20, ?21, ?22, ?23, ?24, ?25, ?26) + ?15, ?16, ?17, ?18, ?19, ?20, ?21, ?22, ?23, ?24, ?25, ?26, ?27) ON CONFLICT(thread_id, started_at) DO UPDATE SET finished_at = excluded.finished_at, harness = COALESCE(excluded.harness, turns.harness), @@ -2171,7 +2220,8 @@ fn write_turn_in(transaction: &Connection, turn: &TurnRow) -> Result<(), String> rustcode_version = COALESCE(excluded.rustcode_version, turns.rustcode_version), diff_files = COALESCE(excluded.diff_files, turns.diff_files), diff_added = COALESCE(excluded.diff_added, turns.diff_added), - diff_removed = COALESCE(excluded.diff_removed, turns.diff_removed)", + diff_removed = COALESCE(excluded.diff_removed, turns.diff_removed), + error_kind = excluded.error_kind", params![ turn.thread_id, turn.started_at, @@ -2199,6 +2249,7 @@ fn write_turn_in(transaction: &Connection, turn: &TurnRow) -> Result<(), String> turn.diff_files, turn.diff_added, turn.diff_removed, + turn.error_kind, ], ) .map_err(|error| error.to_string())?; @@ -2233,14 +2284,20 @@ fn write_turn_in(transaction: &Connection, turn: &TurnRow) -> Result<(), String> let mcp_server = call .mcp_server .clone() - .or_else(|| mcp_server_of(&call.tool_name).map(str::to_owned)); + .or_else(|| mcp_server_of(&call.tool_name).map(str::to_owned)) + .or_else(|| { + call.harness_tool + .as_deref() + .and_then(mcp_server_of) + .map(str::to_owned) + }); let command_name = call.command_name.clone(); transaction .execute( "INSERT INTO tool_calls (thread_id, turn_id, tool_name, success, exit_code, at, started_at, duration_ms, output_bytes, truncated, mcp_server, - command_name, error_kind) - VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12, ?13)", + command_name, error_kind, harness_tool) + VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12, ?13, ?14)", params![ turn.thread_id, turn_id, @@ -2255,6 +2312,7 @@ fn write_turn_in(transaction: &Connection, turn: &TurnRow) -> Result<(), String> mcp_server, command_name, call.error_kind, + call.harness_tool, ], ) .map_err(|error| error.to_string())?; @@ -2480,6 +2538,89 @@ fn parse_numstat(text: &str) -> (i64, i64, i64) { (files, added, removed) } +/// Why a turn failed, as a short kind from its error text. The text itself +/// is never stored: it can hold paths, prompts or tokens. +pub fn turn_failure_kind(error: &str) -> String { + let text = error.to_ascii_lowercase(); + let has = |needles: &[&str]| needles.iter().any(|needle| text.contains(needle)); + // An HTTP status only as a status, never as digits inside an id. + let status = |codes: &[u16]| { + codes.iter().any(|code| { + ["http ", "status ", "status code ", "status: ", "("] + .iter() + .any(|prefix| text.contains(&format!("{prefix}{code}"))) + }) + }; + let kind = if has(&[ + "unknown acp session", + "session not found", + "no such session", + ]) { + "unknown_session" + } else if status(&[429]) || has(&["rate limit", "rate_limit", "too many requests"]) { + "rate_limited" + } else if has(&[ + "usage limit", + "quota", + "credit balance", + "insufficient_quota", + ]) { + "usage_limit" + } else if status(&[401, 403]) + || has(&[ + "unauthorized", + "forbidden", + "not logged in", + "authentication", + "token request failed", + "invalid api key", + ]) + { + "auth" + } else if has(&[ + "context window", + "context length", + "prompt is too long", + "too many tokens", + ]) { + "context_overflow" + } else if status(&[500, 502, 503, 504, 529]) + || has(&["overloaded", "internal server error", "server error"]) + { + "provider_error" + } else if has(&["timed out", "timeout"]) { + "timeout" + } else if has(&[ + "exited", + "broken pipe", + "connection closed", + "process ended", + "killed", + "signal", + ]) { + "harness_exited" + } else if has(&[ + "failed to start", + "spawn", + "not found in path", + "no such file", + ]) { + "harness_start" + } else if has(&[ + "tls", + "certificate", + "unknownissuer", + "dns", + "connection refused", + "network", + ]) { + "network" + } else { + "other" + }; + kind.to_owned() +} + /// The server in an `mcp____` tool name, if it is one. fn mcp_server_of(tool_name: &str) -> Option<&str> { let rest = tool_name.strip_prefix("mcp__")?; @@ -5206,7 +5347,8 @@ mod tests { "truncated", "mcp_server", "command_name", - "error_kind" + "error_kind", + "harness_tool" ] ); drop(first); @@ -6462,6 +6604,104 @@ mod tests { assert_eq!(latency["mean_duration_ms"], 9_500.0); } + #[test] + fn harness_tools_and_failure_kinds_round_trip() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join(DATA_FILE); + let connection = open(&path).unwrap(); + let mut row = turn("t", "2026-11-01T10:00:00Z", (100, 20), "failed"); + row.error_kind = Some(turn_failure_kind("unknown ACP session: 01a1")); + row.tool_calls[0].tool_name = "other".to_owned(); + row.tool_calls[0].harness_tool = Some("mcp__rustcode__wait_session".to_owned()); + write_turn(&connection, &row).unwrap(); + + let stored = one_row(&path, "SELECT error_kind FROM turns"); + assert_eq!(stored["error_kind"], "unknown_session"); + let tool = one_row( + &path, + "SELECT tool_name, harness_tool, mcp_server FROM tool_calls", + ); + assert_eq!(tool["tool_name"], "other"); + assert_eq!(tool["harness_tool"], "mcp__rustcode__wait_session"); + assert_eq!(tool["mcp_server"], "rustcode"); + } + + #[test] + fn turn_failure_kinds_read_the_error_without_keeping_it() { + assert_eq!( + turn_failure_kind("HTTP 429 Too Many Requests"), + "rate_limited" + ); + assert_eq!( + turn_failure_kind("OpenAI token request failed (HTTP 400)"), + "auth" + ); + assert_eq!(turn_failure_kind("status 503 overloaded"), "provider_error"); + assert_eq!( + turn_failure_kind("the harness process exited"), + "harness_exited" + ); + assert_eq!(turn_failure_kind("TLS UnknownIssuer"), "network"); + assert_eq!(turn_failure_kind("something odd"), "other"); + // Digits inside an id are not a status. + assert_eq!( + turn_failure_kind("thread 01a1d4291-4030 went away"), + "other" + ); + } + + #[test] + fn captured_air_compactions_are_backfilled_once() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join(DATA_FILE); + let connection = open(&path).unwrap(); + let payload = serde_json::json!({ + "jsonrpc": "2.0", + "method": "session/update", + "params": {"sessionId": "s", "update": { + "sessionUpdate": "tool_call_update", + "toolCallId": "c1", + "_meta": {"jetbrains": {"air": {"contextCompaction": { + "version": 1, "trigger": "manual", "preTokens": 200_000, + "postTokens": 15_000, "durationMs": 30_000 + }}}} + }} + }); + connection + .execute( + "INSERT INTO harness_rpc_messages (thread_id, harness, direction, message_kind, + method, at, payload_json) + VALUES ('t', 'claude-code', 'harness_to_rustcode', 'notification', + 'session/update', '2026-11-14T22:24:13.712Z', ?1)", + [payload.to_string()], + ) + .unwrap(); + connection + .execute("DELETE FROM schema_migrations WHERE id = 13", []) + .unwrap(); + migrate(&connection).unwrap(); + // Running it again (a second open) adds nothing. + connection + .execute("DELETE FROM schema_migrations WHERE id = 13", []) + .unwrap(); + migrate(&connection).unwrap(); + + let rows = query_path( + &path, + "SELECT trigger, tokens_before, tokens_after, duration_ms, external_id, success FROM compactions", + 10, + ) + .unwrap() + .rows; + assert_eq!(rows.len(), 1); + assert_eq!(rows[0]["trigger"], "manual"); + assert_eq!(rows[0]["tokens_before"], 200_000); + assert_eq!(rows[0]["tokens_after"], 15_000); + assert_eq!(rows[0]["duration_ms"], 30_000); + assert_eq!(rows[0]["external_id"], "c1"); + assert_eq!(rows[0]["success"], 1); + } + /// A thread, a turn and a tool call written with plain SQL against only /// the columns migrations 1 to 8 know, so a test can build an old store. const OLD_ROWS: &str = "INSERT INTO threads (id, title) VALUES ('t', 'kept'); diff --git a/rustcode/engine/src/harness/codex.rs b/rustcode/engine/src/harness/codex.rs index 94bf619ff..ad6a69d6f 100644 --- a/rustcode/engine/src/harness/codex.rs +++ b/rustcode/engine/src/harness/codex.rs @@ -912,7 +912,12 @@ fn is_agent_thread(state: &Mutex, params: &Value) -> bool { /// its steps under it. Any other call is left as it is. fn fold_under_agent(state: &Mutex, params: &Value, call: ToolCall) -> ToolCall { match agent_row_of_call(state, params) { - Some(parent) => call.meta(step_meta(&parent)), + Some(parent) => { + // Keep the tool's own name (`rustcode.toolName`) beside the fold. + let mut meta = call.meta.clone().unwrap_or_default(); + meta.extend(step_meta(&parent)); + call.meta(meta) + } None => call, } } @@ -1094,7 +1099,28 @@ fn tool_call_for(item: &Value) -> Option { .kind(ToolKind::Read), _ => return None, }; - Some(call.status(ToolCallStatus::InProgress)) + let mut meta = Map::new(); + meta.insert( + "rustcode".to_owned(), + json!({"toolName": codex_tool_name(item)}), + ); + Some(call.status(ToolCallStatus::InProgress).meta(meta)) +} + +/// Codex's own name for a tool item, for the data store's `harness_tool`: the +/// item type, `mcp____` for an MCP call (the shape Claude Code +/// uses, so one query covers both), and the tool of a dynamic call. +fn codex_tool_name(item: &Value) -> String { + let kind = str_field(item, "type").unwrap_or("tool"); + match kind { + "mcpToolCall" => format!( + "mcp__{}__{}", + str_field(item, "server").unwrap_or("mcp"), + str_field(item, "tool").unwrap_or("tool") + ), + "dynamicToolCall" => str_field(item, "tool").unwrap_or(kind).to_owned(), + _ => kind.to_owned(), + } } /// The update a finished Codex tool item makes to its card. diff --git a/rustcode/engine/src/harness/codex_tests.rs b/rustcode/engine/src/harness/codex_tests.rs index f73a85c8c..f880c5a67 100644 --- a/rustcode/engine/src/harness/codex_tests.rs +++ b/rustcode/engine/src/harness/codex_tests.rs @@ -1177,7 +1177,8 @@ fn an_agents_own_steps_fold_under_its_row() { json!({"type": "commandExecution", "id": "cmd10", "command": "ls"}), ), ); - assert_eq!(own[0]["_meta"], Value::Null); + assert_eq!(own[0]["_meta"]["claudeCode"], Value::Null); + assert_eq!(own[0]["_meta"]["rustcode"]["toolName"], "commandExecution"); // What the agent says, plans and spends stays out of the thread itself. for (method, params) in [ ( diff --git a/rustcode/engine/src/harness/session.rs b/rustcode/engine/src/harness/session.rs index 30ba5cb1d..7cbb4d379 100644 --- a/rustcode/engine/src/harness/session.rs +++ b/rustcode/engine/src/harness/session.rs @@ -1571,11 +1571,21 @@ impl ExternalSession { ), _ => return None, }; + // The newer adapter marks the call with + // `_meta.jetbrains.air.contextCompaction` and sends its facts (trigger, + // tokens, duration) there, in an update of their own after the + // `completed` status, with no status or title. + let air = meta + .and_then(|meta| meta.get("jetbrains")) + .and_then(|jetbrains| jetbrains.get("air")) + .and_then(|air| air.get("contextCompaction")); + let air_facts = + air.filter(|facts| facts.get("trigger").is_some() || facts.get("preTokens").is_some()); let completed = matches!( status, Some(ToolCallStatus::Completed | ToolCallStatus::Failed) ); - if !completed { + if !completed && air_facts.is_none() { return None; } let named_compaction = meta @@ -1583,9 +1593,10 @@ impl ExternalSession { .and_then(|claude| claude.get("toolName")) .and_then(serde_json::Value::as_str) == Some("compact"); - if !named_compaction && title != Some("Compact conversation") { + if !named_compaction && air.is_none() && title != Some("Compact conversation") { return None; } + let facts = air_facts.or(facts); let number = |name: &str| { facts .and_then(|facts| facts.get(name)) @@ -1654,6 +1665,7 @@ impl ExternalSession { .enumerate() .filter_map(|(offset, message)| { let metrics = transcript.tool_metrics.get(&(history_from + offset)); + let native = transcript.tool_names.get(&(history_from + offset)); let recorded = message.tool_result.as_ref()?; if recorded.pending { return None; @@ -1668,6 +1680,7 @@ impl ExternalSession { output_bytes: Some(message.content.len() as i64), truncated: Some(recorded.truncated || recorded.payload_truncated), mcp_server: None, + harness_tool: native.cloned(), command_name: recorded .command .as_deref() @@ -1774,6 +1787,10 @@ impl ExternalSession { stop_reason: Some(stop_reason.to_owned()), effort: record.effort.clone(), account: record.account.clone(), + error_kind: result + .as_ref() + .err() + .map(|error| crate::data_store::turn_failure_kind(error)), tool_calls, ..Default::default() }, @@ -2033,6 +2050,40 @@ mod launch_tests { ); } + #[test] + fn the_adapters_air_compaction_facts_are_a_row() { + // The newer adapter: the call starts with an empty marker, completes + // with a bare status, then sends its facts in an update of their own. + let started = session_update(serde_json::json!({ + "sessionUpdate": "tool_call", + "toolCallId": "compact-9", + "title": "Compact conversation", + "kind": "think", + "status": "in_progress", + "_meta": {"jetbrains": {"air": {"version": 1, "contextCompaction": {"version": 1}}}}, + })); + assert!(ExternalSession::harness_compaction(&started, "thread-1", "claude-code").is_none()); + let facts = session_update(serde_json::json!({ + "sessionUpdate": "tool_call_update", + "toolCallId": "compact-9", + "_meta": {"jetbrains": {"air": {"contextCompaction": { + "version": 1, + "trigger": "manual", + "preTokens": 213_555, + "postTokens": 14_247, + "durationMs": 36_972, + }}}}, + })); + let row = ExternalSession::harness_compaction(&facts, "thread-1", "claude-code") + .expect("the facts update is the row"); + assert_eq!(row.trigger, "manual"); + assert_eq!(row.tokens_before, Some(213_555)); + assert_eq!(row.tokens_after, Some(14_247)); + assert_eq!(row.duration_ms, Some(36_972)); + assert_eq!(row.external_id.as_deref(), Some("compact-9")); + assert!(row.success); + } + #[test] fn a_compaction_still_running_or_another_tool_is_not_a_row() { let running = session_update(serde_json::json!({ diff --git a/rustcode/engine/src/harness/transcript.rs b/rustcode/engine/src/harness/transcript.rs index 73141af9a..54fe31c1c 100644 --- a/rustcode/engine/src/harness/transcript.rs +++ b/rustcode/engine/src/harness/transcript.rs @@ -55,6 +55,10 @@ pub(crate) struct Transcript { reply: String, live: bool, pub tool_metrics: HashMap, + /// The harness's own name for each tool call (`Bash`, + /// `mcp__rustcode__wait_session`, a Codex item type), by `tool` message + /// index. The message keeps the ACP kind; this is for the data store. + pub tool_names: HashMap, thought: String, /// Tool call id → where its two messages are. tools: HashMap, @@ -235,6 +239,10 @@ impl Transcript { // card text of its own. message.content = text; } + let failure_text = (fields.status == Some(ToolCallStatus::Failed)) + .then(|| fields.content.as_deref().map(summary)) + .flatten(); + let exit_code = exit_code.or_else(|| failure_text.as_deref().and_then(text_exit_code)); let record = message.tool_result.get_or_insert_with(Default::default); if let Some(command) = fields.raw_input.as_ref().and_then(|input| { crate::command_guard::requested_command(input) @@ -252,6 +260,20 @@ impl Transcript { if let Some(code) = exit_code { record.exit_code = Some(code as i32); } + match fields.status { + Some(ToolCallStatus::Failed) => { + record.error_kind = Some(failure_kind( + meta, + record.exit_code, + failure_text.as_deref(), + )); + } + Some(ToolCallStatus::Completed) => record.error_kind = None, + _ => {} + } + if let Some(name) = meta.and_then(native_tool_name) { + self.tool_names.insert(slot.result, name); + } if let Some(locations) = &fields.locations { record.changed_paths = locations .iter() @@ -283,6 +305,7 @@ impl Transcript { pub(crate) fn push_user(&mut self, text: &str) { if !self.live { self.tool_metrics.clear(); + self.tool_names.clear(); } self.end_reply(); self.plan = None; @@ -367,6 +390,87 @@ fn terminal_exit(meta: &Meta) -> Option { .and_then(Value::as_i64) } +/// The harness's own name for a tool call, from its `_meta`: Claude Code's +/// `claudeCode.toolName`, the ACP adapter's `jetbrains.air.mcp` server and +/// tool, or `rustcode.toolName` that the Codex bridge sets. +pub(crate) fn native_tool_name(meta: &Meta) -> Option { + let named = |value: Option<&Value>| { + value + .and_then(Value::as_str) + .map(str::trim) + .filter(|name| !name.is_empty()) + .map(str::to_owned) + }; + if let Some(name) = named( + meta.get("claudeCode") + .and_then(|claude| claude.get("toolName")), + ) { + return Some(name); + } + if let Some(name) = named( + meta.get("rustcode") + .and_then(|rustcode| rustcode.get("toolName")), + ) { + return Some(name); + } + let mcp = meta.get("jetbrains")?.get("air")?.get("mcp")?; + let server = named(mcp.get("server"))?; + let tool = named(mcp.get("tool"))?; + Some(format!("mcp__{server}__{tool}")) +} + +/// Why a tool call failed, as a short kind for the data store; never the +/// message itself. +/// +/// - `permission_denied`: the harness or its permission check refused it +/// (Claude Code's `toolResponse.decisionReasonType`, a "Permission denied" or +/// "user rejected" result). +/// - `guard_refused`: RustCode's command guard refused it. +/// - `exit_status`: a command that ran and exited non-zero. +/// - `timeout`: the harness reports it timed out. +/// - `failed`: anything else. +fn failure_kind(meta: Option<&Meta>, exit_code: Option, text: Option<&str>) -> String { + let decided = meta + .and_then(|meta| meta.get("claudeCode")) + .and_then(|claude| claude.get("toolResponse")) + .and_then(|response| response.get("decisionReasonType")) + .is_some(); + let text = text.unwrap_or_default(); + let head: String = text + .chars() + .take(400) + .collect::() + .to_ascii_lowercase(); + let kind = if head.contains("command guard") { + "guard_refused" + } else if decided + || head.contains("permission denied") + || head.contains("user rejected") + || head.contains("user denied") + || head.contains("doesn't want to proceed") + { + "permission_denied" + } else if exit_code.is_some_and(|code| code != 0) { + "exit_status" + } else if head.contains("timed out") || head.contains("timeout") { + "timeout" + } else { + "failed" + }; + kind.to_owned() +} + +/// The exit code Claude Code writes at the head of a failed Bash result +/// (`Exit code 1`). +fn text_exit_code(text: &str) -> Option { + let text = text.trim_start().trim_start_matches("```").trim_start(); + text.strip_prefix("Exit code ")? + .split(|ch: char| !ch.is_ascii_digit() && ch != '-') + .next()? + .parse() + .ok() +} + /// The exit code a structured `rawOutput` carries (`exitCode` or /// `exit_code`), for a harness that does not use terminal meta. fn raw_exit_code(raw: &Value) -> Option { @@ -479,6 +583,9 @@ fn finalize_tool_message(message: &mut ChatMessage) { { record.pending = false; record.success = false; + record + .error_kind + .get_or_insert_with(|| "interrupted".to_owned()); } if message.content.is_empty() { return; @@ -627,6 +734,110 @@ mod tests { use super::*; use agent_client_protocol::schema::v1::ToolCallUpdate; + #[test] + fn a_failed_claude_bash_call_keeps_its_name_exit_code_and_kind() { + let mut transcript = Transcript::default(); + transcript.push_user("go"); + let mut meta = Meta::new(); + meta.insert( + "claudeCode".to_owned(), + serde_json::json!({"toolName": "Bash"}), + ); + transcript.observe(&SessionUpdate::ToolCall( + ToolCall::new("t", "cargo check") + .kind(ToolKind::Execute) + .raw_input(serde_json::json!({"command": "cd repo && cargo check"})) + .status(ToolCallStatus::InProgress) + .meta(meta), + )); + transcript.observe(&SessionUpdate::ToolCallUpdate(ToolCallUpdate::new( + "t", + ToolCallUpdateFields::new() + .status(ToolCallStatus::Failed) + .content(vec![ToolCallContent::from(ContentBlock::Text( + TextContent::new("```\nExit code 101\nerror[E0425]\n```"), + ))]), + ))); + let index = transcript.history.len() - 1; + let record = transcript.history[index].tool_result.as_ref().unwrap(); + assert_eq!(record.tool_name, "execute"); + assert_eq!(record.exit_code, Some(101)); + assert_eq!(record.error_kind.as_deref(), Some("exit_status")); + assert_eq!( + transcript.tool_names.get(&index).map(String::as_str), + Some("Bash") + ); + } + + #[test] + fn mcp_names_and_failure_kinds_come_from_meta_and_text() { + let mut meta = Meta::new(); + meta.insert( + "jetbrains".to_owned(), + serde_json::json!({"air": {"mcp": {"server": "rustcode", "tool": "wait_session"}}}), + ); + assert_eq!( + native_tool_name(&meta).as_deref(), + Some("mcp__rustcode__wait_session") + ); + let mut codex = Meta::new(); + codex.insert( + "rustcode".to_owned(), + serde_json::json!({"toolName": "commandExecution"}), + ); + assert_eq!( + native_tool_name(&codex).as_deref(), + Some("commandExecution") + ); + assert_eq!(native_tool_name(&Meta::new()), None); + + let mut denied = Meta::new(); + denied.insert( + "claudeCode".to_owned(), + serde_json::json!({"toolResponse": {"decisionReasonType": "classifier"}}), + ); + assert_eq!( + failure_kind(Some(&denied), None, Some("no verdict")), + "permission_denied" + ); + assert_eq!( + failure_kind( + None, + None, + Some("Command guard: rule `no-git-stash` refused") + ), + "guard_refused" + ); + assert_eq!( + failure_kind(None, Some(2), Some("Exit code 2")), + "exit_status" + ); + assert_eq!(failure_kind(None, None, Some("something broke")), "failed"); + assert_eq!(text_exit_code("Exit code 1\nboom"), Some(1)); + assert_eq!(text_exit_code("no code here"), None); + } + + #[test] + fn a_call_cut_off_by_the_turn_end_is_interrupted() { + let mut transcript = Transcript::default(); + transcript.push_user("go"); + transcript.observe(&SessionUpdate::ToolCall( + ToolCall::new("t", "sleep 60") + .kind(ToolKind::Execute) + .status(ToolCallStatus::InProgress), + )); + transcript.end_reply(); + let record = transcript + .history + .last() + .unwrap() + .tool_result + .as_ref() + .unwrap(); + assert!(!record.success); + assert_eq!(record.error_kind.as_deref(), Some("interrupted")); + } + #[test] fn live_tool_metrics_keep_first_terminal_and_command() { let mut transcript = Transcript::default(); diff --git a/rustcode/engine/src/network/turn/finish.rs b/rustcode/engine/src/network/turn/finish.rs index 0cdeda5d8..5d7a5bf82 100644 --- a/rustcode/engine/src/network/turn/finish.rs +++ b/rustcode/engine/src/network/turn/finish.rs @@ -396,6 +396,7 @@ fn record_data_store_turn( output_bytes: Some(message.content.len() as i64), truncated: Some(result.truncated || result.payload_truncated), mcp_server: None, + harness_tool: None, command_name: result .command .as_deref() diff --git a/rustcode/engine/src/tools/misc.rs b/rustcode/engine/src/tools/misc.rs index e5c478910..45f146d0c 100644 --- a/rustcode/engine/src/tools/misc.rs +++ b/rustcode/engine/src/tools/misc.rs @@ -1144,7 +1144,7 @@ pub const QUERY_DATA: Tool = Tool { description: concat!( "Run one read-only SQL statement (SQLite) over what RustCode recorded, for questions thread_usage cannot answer: which tools fail, an exit code, a day's turns. Returns columns and rows, at most ", "1000 rows. Ask it when the user asks something about their own history. ", - "Tables: threads(id, title, project, harness, model, created_at); turns(thread_id, started_at, finished_at, harness, model, input_tokens, output_tokens, cost_usd, tool_calls, status, duration_ms, context_tokens, context_window, stop_reason); approvals(thread_id, tool_name, decision); tool_calls(thread_id, turn_id, tool_name, success, exit_code, at, duration_ms, output_bytes, truncated, mcp_server, command_name, error_kind); compactions(thread_id, harness, model, at, duration_ms, trigger, mode, tokens_before, tokens_after, count, success, source), trigger is auto_context, manual, cache_expiry or harness_auto; thread_links(...); daily_usage, daily_compactions, tool_failure_rates and turn_latency_daily views; provider_requests(thread_id, started_at, status_code, error_kind, request_payload_json); harness_rpc_messages(thread_id, direction, method, at, latency_ms, payload_json); operational_events(thread_id, sequence, event_type, at, fields_json); store_meta internal. Structured rows are kept forever; raw payload bodies are NULL unless captured. Tokens and cost_usd are NULL when unreported, never guessed. This tool needs \"Let agents read threads in other projects\" on, since SQL reads every project." + "Tables: threads(id, title, project, harness, model, created_at); turns(thread_id, started_at, finished_at, harness, model, input_tokens, output_tokens, cached_tokens, cache_write_tokens, cost_usd, tool_calls, status, duration_ms, context_tokens, context_window, stop_reason, error_kind), Codex input includes cached; approvals(thread_id, tool_name, decision); tool_calls(thread_id, turn_id, tool_name, harness_tool, success, exit_code, at, duration_ms, output_bytes, truncated, mcp_server, command_name, error_kind), harness_tool is the harness's own name; compactions(thread_id, harness, model, at, duration_ms, trigger, mode, tokens_before, tokens_after, count, success, source), trigger is auto_context, manual, cache_expiry or harness_auto; thread_links(...); daily_usage, daily_compactions, tool_failure_rates and turn_latency_daily views; provider_requests(thread_id, started_at, status_code, error_kind, request_payload_json); harness_rpc_messages(thread_id, direction, method, at, latency_ms, payload_json); operational_events(thread_id, sequence, event_type, at, fields_json); store_meta internal. Raw payloads are NULL unless captured. Tokens and cost_usd are NULL when unreported, never guessed. Needs \"Let agents read threads in other projects\" on." ), arguments: r#"{"sql": "SELECT tool_name, COUNT(*) FROM tool_calls WHERE success = 0 GROUP BY 1 ORDER BY 2 DESC", "limit": 50}"#, handler: orchestration_unavailable,