Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ Two directors, selected by role:
Auto mode is toggled by CLI flags (`--auto` / `--no-auto`); there is currently no in-session key to toggle it (default on; constrained envelope — workspace writes and unconstrained shell auto-allow; installs, recursive rm, force/uncontained worktree changes, sensitive-path and opaque-wrapper shell still ask; contained non-force `git worktree add`/`remove`/`prune` and `list` auto-allow; shell file-mutation denied). It is not a separate edit/plan mode.

- **SubAgentDirector** (delegated work, `src/subagent/index.ts`) — Drives a dispatched worker until a turn arrives with no tool calls, then replies with the final assistant text and ends the run. A tool-less turn **after tools** completes only with the four-heading envelope (Summary, Findings, Blockers, Paths). Assistant text that prints explicit `<tool_call>` markup is treated as attempted tool use, not narration: one **verbatim-tool-call** nudge asks the worker to re-issue a real `tool_call` and does not count toward the tool-less spiral. A missing envelope otherwise nudges once (**incomplete-report**) and a second tool-less turn still without the envelope salvages as **incomplete-report-stop** (once; later tool-less turns wait). Explore/read-only workers that used tools then replied with findings remain normal completes; `requireEvidence` (off by default, set per director) additionally requires at least one read before a tool-less spawn-only reply can complete. `requirePlanSubstance` (planner or `intent=plan`, not `modelRole === "plan"`) additionally requires Findings to contain files/paths, acceptance criteria, non-goals, risks, and ordered steps with a non-placeholder line each — four headings with stub Findings are incomplete-report, not an attachable plan. After real tool work, wrap-up Findings that are not placeholder or outline-only complete instead of being salvaged as a stub plan. Reads done through `run_shell` count as evidence too — `src/subagent/shell-evidence.ts` classifies shell reads (`cat`, `grep`, `sed` without `-i`, …) over the same subject expansion the auto-shell policy uses — but there is no corresponding shell-write evidence or file-write requirement: a run that never touches a file still completes normally once it replies with the envelope. There is no turn budget. Operator/parent cancel after any progress returns a **cancelled** salvage report (partial findings + tool activity) instead of a bare cancel string; cancel before progress still surfaces as cancelled-by-operator. There is no repetition/no-progress/never-acted/never-edited hard stop and no fingerprint-based re-dispatch block — a genuinely stuck worker runs until it completes, stalls, hits an opt-in wall-clock deadline, or is cancelled.
`spawn_agent` starts each worker and records it in the caller's fleet mailbox. On the TUI primary, mailbox mail is the collect path: occupancy takes uncollected terminals and re-enters the parent as system inbound with a Summary + Blockers digest (Findings when Summary is empty) and a blob pointer named for `read_file`. Nested orchestrators collect through mailbox mail the same way. A mounted `wait_agents` (exec primary) collects worker reports directly instead. Already-collected waits return status without a second report or error body. Wait JSON includes `stop_reason` from the session when present so a salvage that is wait-`done` is not mistaken for a clean complete, and so parent-initiated interrupt (`interrupted`) is not mistaken for operator-cancel (`cancelled`). Deadline salvage prepends an advisory parent hint suggesting continuation plus a longer deadline if more wall-clock time is warranted. Failed and incomplete-report salvage tell the parent to diagnose from the report or error and MAY spawn one successor with a changed brief. A parent-initiated interrupt is a resumable pause: wait unblocks with `stop_reason: interrupted` (often while the session is still running and has no report); the parent should `resume_agent` or re-wait, and must not spawn a successor against a still-live worker. Successor only if that session is no longer resumable. Operator-cancelled salvage asks the parent to synthesize Findings and Paths and wait for the operator instead of auto-starting another specialist. Identical re-dispatch of the same brief stays refused at the prompt / spawn-handoff layer, except a recoverable/continuable child failure (`continuable: true`) MAY spawn one successor with the same brief; there is no fingerprint-based re-dispatch hard-block. Deadline hints are advisory only — an identical re-dispatch is still admitted at runtime. Parent hints are prepended on salvage reports returned to the parent. The runtime does not auto-spawn successors.
`spawn_agent` starts each worker and records it in the caller's fleet mailbox. On the TUI primary, mailbox mail is the collect path: occupancy takes uncollected terminals and re-enters the parent as system inbound with a Summary + Blockers digest (Findings when Summary is empty) and a blob pointer named for `read_file`. Nested orchestrators collect through mailbox mail the same way. A mounted `wait_agents` (exec primary) collects worker reports directly instead. Already-collected waits return status without a second report or error body. Wait JSON includes `stop_reason` from the session when present so a salvage that is wait-`done` is not mistaken for a clean complete, and so parent-initiated interrupt (`interrupted`) is not mistaken for operator-cancel (`cancelled`). Every terminal non-clean outcome (failed, cancelled, interrupted, or a forced-stop salvage) also carries a `failure` record, written in the same store mutation that makes the outcome visible and never rewritten: a typed `failure_class` (`provider_retryable`, `provider_fatal`, `loop_guard` for the reactor's doom-loop guard, the forced-stop reasons, or `error`), `recovery` availability with a typed reason, the attempt number, the ask_director `handoff` state at termination (`delivered` only when the worker's own ask_director call returned the answer), and whether `cleanup` released everything. `provider_failure` and `continuable` derive from that record, so wait_agents, list_agents, and mailbox mail cannot disagree. Deadline salvage prepends an advisory parent hint suggesting continuation plus a longer deadline if more wall-clock time is warranted. Failed and incomplete-report salvage tell the parent to diagnose from the report or error and MAY spawn one successor with a changed brief. A parent-initiated interrupt is a resumable pause: wait unblocks with `stop_reason: interrupted` (often while the session is still running and has no report); the parent should `resume_agent` or re-wait, and must not spawn a successor against a still-live worker. Successor only if that session is no longer resumable. Operator-cancelled salvage asks the parent to synthesize Findings and Paths and wait for the operator instead of auto-starting another specialist. Identical re-dispatch of the same brief stays refused at the prompt / spawn-handoff layer, except a recoverable/continuable child failure (`continuable: true`) MAY spawn one successor with the same brief; there is no fingerprint-based re-dispatch hard-block. Deadline hints are advisory only — an identical re-dispatch is still admitted at runtime. Parent hints are prepended on salvage reports returned to the parent. The runtime does not auto-spawn successors.

#### Model-family policy (`src/agent/model-family-policy.ts`)

Expand Down
24 changes: 21 additions & 3 deletions src/subagent/agent-fleet.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1227,9 +1227,15 @@ describe("interrupt_agent unblocks wait_agents", () => {
agent_id: string;
status: string;
stop_reason?: string;
failure?: unknown;
}[];
expect(results).toEqual([
{ agent_id: id, status: "interrupted", stop_reason: "interrupted" },
{
agent_id: id,
status: "interrupted",
stop_reason: "interrupted",
failure: expect.objectContaining({ failure_class: "interrupted" }),
},
]);
expect(deps.sessions.get(id)?.lifecycleStatus).toBe("interrupted");
expect(deps.sessions.get(id)?.status).toBe("running");
Expand Down Expand Up @@ -1589,9 +1595,15 @@ describe("interrupt_agent unblocks wait_agents", () => {
agent_id: string;
status: string;
stop_reason?: string;
failure?: unknown;
}[];
expect(results).toEqual([
{ agent_id: id, status: "interrupted", stop_reason: "interrupted" },
{
agent_id: id,
status: "interrupted",
stop_reason: "interrupted",
failure: expect.objectContaining({ failure_class: "interrupted" }),
},
]);
expect(deps.fleetRecords.peek(id)?.status).toBe("interrupted");
expect(deps.fleetRecords.peek(id)?.collected).toBe(true);
Expand Down Expand Up @@ -2520,9 +2532,15 @@ describe("admission queue", () => {
agent_id: string;
status: string;
stop_reason?: string;
failure?: unknown;
}[];
expect(results).toEqual([
{ agent_id: queuedId, status: "interrupted", stop_reason: "cancelled" },
{
agent_id: queuedId,
status: "interrupted",
stop_reason: "cancelled",
failure: expect.objectContaining({ failure_class: "cancelled" }),
},
]);
expect(started()).toBe(1);

Expand Down
78 changes: 40 additions & 38 deletions src/subagent/agent-fleet.ts
Original file line number Diff line number Diff line change
Expand Up @@ -111,9 +111,11 @@ import {

import { formatSubAgentSpawnAuthFailureMessage } from "./inference-auth-failure.js";
import {
isRecoverableProviderFailureCategory,
isResolvedProviderFailureError,
} from "../inference-error-message.js";
classifySubAgentFailure,
isProviderFailureClass,
isSubAgentLoopGuardError,
type SubAgentFailureRecord,
} from "./terminal-failure.js";
import { errorMessage } from "../agent/error-message.js";
import { isSubAgentCancelError } from "./dispose.js";
import {
Expand All @@ -132,9 +134,11 @@ interface FleetRecord {
report?: string;
error?: string;
stopReason?: string;
/** Derived from `failure.failure_class`. */
providerFailure?: true;
/** Transient provider failure — the parent may spawn one successor. */
/** Transient provider failure, derived from `failure.recovery.available`. */
recoverableFailure?: true;
failure?: SubAgentFailureRecord;
/** Set once a wait_agents caller has been handed this result. */
collected?: boolean;
/** Set once a waiter or occupancy take handed report/error. */
Expand Down Expand Up @@ -169,9 +173,8 @@ interface FleetOverlay {
bodyHanded?: boolean;
tombstoned?: boolean;
hint?: string;
providerFailure?: true;
/** Transient provider failure — same meaning as the FleetRecord flag. */
recoverableFailure?: true;
/** Mirror of the session's failure record so it survives eviction. */
failure?: SubAgentFailureRecord;
}

const RECOVERY_HINT =
Expand Down Expand Up @@ -267,20 +270,6 @@ class FleetMailbox {
this.sessions.wake();
}

markProviderFailure(id: string): void {
const existing = this.records.get(id);
if (existing === undefined) return;
existing.providerFailure = true;
}

/** Stamp a transient provider failure beside sessions.fail; the snapshot
* projects it even once the payload is tombstoned. */
markRecoverable(id: string): void {
const existing = this.records.get(id);
if (existing === undefined) return;
existing.recoverableFailure = true;
}

markQueued(id: string): void {
const existing = this.records.get(id);
if (existing === undefined || existing.collected === true) return;
Expand Down Expand Up @@ -381,6 +370,7 @@ class FleetMailbox {

private rememberLiveWaitStatuses(): void {
for (const [id, overlay] of this.records) {
this.mirrorFailure(id, overlay);
const wait = this.sessionWaitStatus(id);
if (wait === undefined) continue;
// Keep lastWait while admission-queued; the live projection would
Expand All @@ -390,6 +380,13 @@ class FleetMailbox {
}
}

private mirrorFailure(id: string, overlay: FleetOverlay): void {
const session = this.sessions.get(id);
if (session === undefined) return;
if (session.failure !== undefined) overlay.failure = session.failure;
else delete overlay.failure;
}

private waitStatusFromEvicted(id: string): WaitJSONStatus | undefined {
return waitStatusFromVerbLifecycle(this.sessions.evictedLifecycle(id));
}
Expand Down Expand Up @@ -430,6 +427,7 @@ class FleetMailbox {
return { status: "running" };
}
const session = this.sessions.get(id);
this.mirrorFailure(id, overlay);
if (
session === undefined &&
overlay.tombstoned !== true &&
Expand Down Expand Up @@ -457,6 +455,7 @@ class FleetMailbox {
: payload?.stopReason;
const ask =
status === "awaiting_director" ? this.sessions.peekAsk(id) : undefined;
const failure = isLiveWaitStatus(status) ? undefined : overlay.failure;
return {
status,
...(overlay.collected === true ? { collected: true } : {}),
Expand All @@ -468,10 +467,13 @@ class FleetMailbox {
? { error: payload.error }
: {}),
...(stopReason !== undefined ? { stopReason } : {}),
...(overlay.providerFailure === true ? { providerFailure: true } : {}),
...(overlay.recoverableFailure === true
...(failure !== undefined && isProviderFailureClass(failure.failure_class)
? { providerFailure: true }
: {}),
...(failure?.recovery.available === true
? { recoverableFailure: true }
: {}),
...(failure !== undefined ? { failure } : {}),
...(ask !== undefined
? {
question: ask.question,
Expand Down Expand Up @@ -1409,7 +1411,7 @@ export function createSpawnAgentTool(deps: AgentFleetDeps): AgentTool {
log.error("spawn_agent worktree setup failed: {error}", {
error: message,
});
deps.sessions.fail(session.id, message);
await deps.sessions.fail(session.id, message);
finalizeEnd(true);
admission.release(session.id);
return;
Expand Down Expand Up @@ -1660,7 +1662,7 @@ export function createSpawnAgentTool(deps: AgentFleetDeps): AgentTool {
: {}),
});
})
.catch((err) => {
.catch(async (err) => {
const alreadyCancelled =
deps.sessions.get(session.id)?.status === "cancelled";
if (
Expand All @@ -1673,25 +1675,24 @@ export function createSpawnAgentTool(deps: AgentFleetDeps): AgentTool {
deps.sessions.settleRun(session.id);
return;
}
const isProviderFailure = isResolvedProviderFailureError(err);
const diagnosticMessage = errorMessage(err);
const authMessage = formatSubAgentSpawnAuthFailureMessage(
description,
err,
);
const failReason = authMessage ?? diagnosticMessage;
if (isProviderFailure || providerFailureObserved) {
deps.fleetRecords.markProviderFailure(session.id);
}
// A transient failure stays wait-terminal failed but carries a
// continuable marker so the parent can spawn one successor.
if (
isResolvedProviderFailureError(err) &&
isRecoverableProviderFailureCategory(err.category)
) {
deps.fleetRecords.markRecoverable(session.id);
}
deps.sessions.fail(session.id, failReason);
// CL-10202: the typed class is the single source for the
// provider_failure and continuable projections. A transient
// failure stays wait-terminal failed; nothing here retries.
await deps.sessions.fail(session.id, failReason, {
failure_class: classifySubAgentFailure(err, {
providerFailureObserved,
}),
...(settlement?.teardown_failed === true
? { teardownFailed: true }
: {}),
...(isSubAgentLoopGuardError(err) ? { cause: err.cause } : {}),
});
})
.finally(() => {
activeLanes.delete(call.id);
Expand Down Expand Up @@ -2031,6 +2032,7 @@ export function createListAgentsTool(deps: WaitAgentsDeps): AgentTool {
}
: {}),
...(stopReason !== undefined ? { stop_reason: stopReason } : {}),
...(record?.failure !== undefined ? { failure: record.failure } : {}),
...(record?.status === "awaiting_director" &&
record.question !== undefined
? { question: record.question }
Expand Down
1 change: 1 addition & 0 deletions src/subagent/fleet-dry-drive.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -263,6 +263,7 @@ describe("collectUncollectedTerminals", () => {
status: "failed",
description: "failed lane",
error: "boom",
failure: expect.objectContaining({ failure_class: "error" }),
});
expect(mailbox.peek("done")?.collected).toBe(true);
expect(mailbox.peek("fail")?.collected).toBe(true);
Expand Down
6 changes: 6 additions & 0 deletions src/subagent/fleet-dry-drive.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import {
} from "../plugins/result-truncation-plugin.js";
import { isLiveWaitStatus, type WaitJSONStatus } from "./lifecycle.js";
import { parseSubAgentReport } from "./report.js";
import type { SubAgentFailureRecord } from "./terminal-failure.js";

/** Enough of a lane report for a parent continuation; traces stay on disk. */
export const FLEET_DRY_REPORT_CHARS = 8_192;
Expand Down Expand Up @@ -44,6 +45,7 @@ export interface FleetDryMailboxRecord {
/** Transient provider failure — the parent may spawn one successor. */
readonly recoverableFailure?: true;
readonly stopReason?: string;
readonly failure?: SubAgentFailureRecord;
}

export interface FleetDryMailbox {
Expand Down Expand Up @@ -118,6 +120,7 @@ export interface CollectedWorkerReport {
continuable?: true;
continue_with?: string;
stop_reason?: string;
failure?: SubAgentFailureRecord;
}

/** Mailbox parent payload: envelope digest plus a blob pointer, not the full report. */
Expand All @@ -137,6 +140,7 @@ export interface MailboxWorkerDigest {
continuable?: true;
continue_with?: string;
stop_reason?: string;
failure?: SubAgentFailureRecord;
}

/**
Expand Down Expand Up @@ -344,6 +348,7 @@ export async function digestCollectedReport(
...(report.stop_reason !== undefined
? { stop_reason: report.stop_reason }
: {}),
...(report.failure !== undefined ? { failure: report.failure } : {}),
};
}

Expand Down Expand Up @@ -442,6 +447,7 @@ export function projectMailboxRecord(
...(taken.stopReason !== undefined
? { stop_reason: taken.stopReason }
: {}),
...(taken.failure !== undefined ? { failure: taken.failure } : {}),
};
}

Expand Down
Loading
Loading