Skip to content

test: drop hollow prompt and span pins #1490

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
TheGreatAxios merged 2 commits into from
Oct 17, 2026
Merged

test: drop hollow prompt and span pins #1490

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
26f07f0
test: drop hollow prompt and span pins
TheGreatAxios Oct 17, 2026
acc2469
test: drop the unused prompt-size export
TheGreatAxios Oct 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/ARCHITECTURE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -346,7 +346,7 @@ The primary director is **dispatch** (`buildChatRole` → `createDispatchSystemP
- `buildHarnessFacts` — the non-derivable rules: shell file-writes are blocked, path tools are the DIY surface on primary (spawn coder/shakespeare directors for substantial work), dependency installs and off-limits paths need approval, images are native multimodal input, core tools plus the advertised catalog (including `skill_search`) are resident (MCP and other unadvertised tools load via `tool_search`; use `search_agents` before dispatching specialists), workflows run only from slash-command steps, and session memory lives at `.corbits/MEMORY.md`.
- `buildGuidelines` — be concise, prefer `spawn_agent` (mailbox mail on TUI; exec-primary `wait_agents`) for substantial product work with one focused task per worker and fan-out by independent lanes, DIY tiny/bounded edits on the parent, answer questions and diagnose visual/product feedback before editing, work autonomously for explicit coding tasks, use `lsp` for symbol work, and require implementation agents to run the repository-defined typecheck, relevant tests, and every defined full verification command. Agents report exact commands, outcomes, and exit statuses; a repository with no typecheck command produces an explicit Blocker backed by project-configuration evidence rather than an invented command or silent skip.
- `buildPromptDisciplineBlock` — a shared, prohibition-form section appended exactly once to every built prompt (chat and sub-agent, every provider family). Primary vs worker wording differs for product writes: workers are told to use `read_file`/`edit_file`/`write_file`; dispatch is told to DIY tiny/bounded edits with those path tools and spawn directors for substantial work. Shared rules: never `cat`/`sed`/heredoc/`echo` for file work, no setting or exporting environment variables (recurring needs belong in project settings), `web_fetch`/`web_search` instead of `curl`/`wget`/hand-rolled queries, one operation per `run_shell` call, turn semantics (a tool-less reply is the final answer, no repeat searches, stop and change approach after three failed attempts, batch independent reads in parallel), and TTY output rules (short bold headers, one-line bullets, backticks for paths/commands, no wide tables).
- `@corbits/prompt-fragments` — the only copy of director card prose. Cards compose the identity banner, report-envelope shape, DONE GATE / OUT OF LANE frames, and `DISPATCH_ROUTE_ROWS`; director modules call the composed card and do not restate it. Composed bytes are hash-pinned in `src/agent/directors/prompt-card.test.ts`; route rows are drift-pinned against `DIRECTOR_REGISTRY`.
- `@corbits/prompt-fragments` — the only copy of director card prose. Cards compose the identity banner, report-envelope shape, DONE GATE / OUT OF LANE frames, and `DISPATCH_ROUTE_ROWS`; director modules call the composed card and do not restate it. Registry system prompts are the composed cards (`src/agent/directors/prompt-card.test.ts`); route rows are drift-pinned against `DIRECTOR_REGISTRY`.

**Provider-conditional residuals.** Per-family additions layer on top of the shared block via the same `ModelFamilyPolicy` mechanism the directors use (`src/subagent/provider-family.ts`, `src/agent/model-family-policy.ts`) — additive lines, never prompt forks. **Grok** leaves get `buildGrokLeafAntiThrashNote` (gated by `shouldApplyGrokAntiThrash` / `applyGrokFinishBias`, withheld from orchestrators): a compact finish-bias reinforcement plus a one-line reminder to route file/web work through the dedicated tools rather than `run_shell`, motivated by observed tool-routing thrash on the same harness. **Kimi** intentionally has no residual yet — `detectModelFamily` already resolves the family so callers can branch on it, but the prompt seam is left unfilled pending eval characterization of Kimi's behavior, mirroring the provisional (permissive-default) policy in `model-family-policy.ts`.

Expand Down
58 changes: 0 additions & 58 deletions packages/prompt-variance/src/rows.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,17 +6,10 @@ import {
FAMILY_ROWS,
gptRow,
grokRow,
grokToolBudgetResidual,
museRow,
type PromptVarianceFamily,
} from "./rows.js";

const CEREMONY_LINES = [
"Never run git add, git commit, git stash, or any other state-changing git command unless the user asks.",
"Do not narrate a plan before acting on a small task; act, then report.",
"Verify with the test command once at the end, not after every edit.",
];

describe("prompt-variance family rows", () => {
test("ships exactly the default/muse/grok/claude/gpt families", () => {
expect([...FAMILY_IDS]).toEqual([
Expand Down Expand Up @@ -61,55 +54,4 @@ describe("prompt-variance family rows", () => {
"muse",
]);
});

test("muse row is the shipped CL-7869 tool-discipline text", () => {
expect(museRow.residual).toContain("Tool discipline:");
expect(museRow.residual).toContain("Batch independent tool calls");
expect(museRow.residual).toContain("Never re-read a file");
expect(museRow.residual).toContain("Do not narrate; act.");
});

test("grok row is the single finish-bias + ceremony block (CL-8296)", () => {
expect(grokRow.residual).toContain("Finish bias (xAI / Grok worker):");
expect(grokRow.residual).toContain("prefer the structured report");
expect(grokRow.residual).toContain("re-open paths you already read");
expect(grokRow.residual).toContain("done-definition is met");
expect(grokRow.residual).toContain("never bash");
});

test("each ceremony line appears exactly once in the grok row (P2 invariant)", () => {
for (const line of CEREMONY_LINES) {
const occurrences = grokRow.residual.split(line).length - 1;
expect(occurrences).toBe(1);
}
});

test("grok tool-budget residual is ceremony-free and budget text appears once (P1 invariant)", () => {
expect(grokToolBudgetResidual).toContain("Tool budget:");
for (const line of CEREMONY_LINES) {
expect(grokToolBudgetResidual).not.toContain(line);
}
const budgetOccurrences =
grokToolBudgetResidual.split("Tool budget:").length - 1;
expect(budgetOccurrences).toBe(1);
});

test("default row carries no residual", () => {
expect(defaultRow.residual).toBe("");
});

test("claude row is the single XML task_guidance block (CL-8309)", () => {
expect(claudeRow.residual).toContain("<task_guidance>");
expect(claudeRow.residual).toContain("</task_guidance>");
expect(claudeRow.residual).toContain("Follow the dispatch brief exactly");
expect(claudeRow.residual).toContain("Batch independent tool calls");
});

test("gpt row is the narrate-before-tools nudge (CL-8310)", () => {
expect(gptRow.residual).toContain("Narrate before tools (GPT worker):");
expect(gptRow.residual).toContain(
"one short line saying what you are doing",
);
expect(gptRow.residual).toContain("done-definition is met");
});
});
74 changes: 0 additions & 74 deletions src/agent/canonical-tool-name.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
import { describe, expect, test } from "bun:test";
import { join } from "node:path";
import { isSameTool } from "./canonical-tool-name.js";

describe("isSameTool", () => {
Expand Down Expand Up @@ -50,76 +49,3 @@ describe("isSameTool", () => {
expect(isSameTool(b, a)).toBe(false);
});
});

// Callsites that compare incoming wire names must go through isSameTool (or
// the canonical modules themselves). A new raw `name === "<tool>"` compare in
// one of these files reintroduces the wire-rename breakage on the next rename,
// so this test fails until the callsite is converted.
describe("tool-name comparison regression guard", () => {
const CALLSITES = [
"src/agent/director.ts",
"src/subagent/poll-exempt.ts",
"src/tui/mcp-view.ts",
"src/tui/runtime-bridge.ts",
"src/tui/stall-watchdog.ts",
"src/agent/tool-execution-watchdog.ts",
"src/tui/turns-to-blocks.ts",
];

const TOOL_LITERALS = new Set([
"read_file",
"write_file",
"edit_file",
"delete_file",
"run_shell",
"search_files",
"manage_tasks",
"use_skill",
"web_fetch",
"web_search",
"ask_operator",
"ask_director",
"wait_agents",
"spawn_agent",
"submit_output",
"submit_plan",
"present",
"list_agents",
"read",
"write",
"edit",
"delete",
"bash",
"shell",
"glob",
"todowrite",
"skill",
"webfetch",
"websearch",
"question",
"update_plan",
"wait",
]);

test("no raw tool-name ===/!== compares outside the canonical modules", async () => {
const root = join(import.meta.dirname, "..", "..");
const violations: string[] = [];
for (const file of CALLSITES) {
const text = await Bun.file(join(root, file)).text();
for (const [index, line] of text.split("\n").entries()) {
const match = /(?:\?\.name|\.name)\s*(===|!==)\s*(["'])([^"']+)\2/.exec(
line,
);
if (match?.[3] !== undefined && TOOL_LITERALS.has(match[3])) {
violations.push(`${file}:${index + 1}: ${line.trim()}`);
}
const canonical =
/canonicalToolName\(([^)]+)\)\s*(===|!==)\s*["']/.exec(line);
if (canonical !== null) {
violations.push(`${file}:${index + 1}: ${line.trim()}`);
}
}
}
expect(violations).toEqual([]);
});
});
45 changes: 0 additions & 45 deletions src/agent/directors/prompt-card.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
import { createHash } from "node:crypto";
import { readFileSync } from "node:fs";
import { describe, expect, test } from "bun:test";

import {
Expand All @@ -14,43 +12,13 @@ import {
import { DIRECTOR_REGISTRY } from "./registry.js";
import { DIRECTOR_IDS } from "./types.js";

/**
* Composed-output pin (CL-10674). The fragment package is the only copy of
* the card prose; these hashes fail when a fragment edit changes bytes.
* Director modules must keep calling the composed card, not restating it.
*/
const CARD_SHA256: Record<(typeof DIRECTOR_IDS)[number], string> = {
dispatch: "0ce0493f3119a02b999b87b5e5c51d61df01ae8042923710404cae6ee9758328",
explorer: "d4cac32fe0703953bf4d68c68b02aa67e3e50d2172b8137c5692ccdb1afc76eb",
planner: "6ce04bb2349e9a67cfab538c1cbcb08c4eaa7dfe2eb40dca94bd022e4608bc23",
coder: "6e872830d642935e98407b1e45a23740e73c731abfb3e6297a242201abd09092",
reviewer: "9de0237970a17a030db9bc4c387d46d5d25788e91a806a7e9713b93778d75d5b",
designer: "d021f346f451eda0770d6580b7647441311c89faef919e670ac6ab4ae9aa297f",
artist: "8ba1a961af72fabbaa6310b322e931872aa759e6572dda327f48d9deefe12ecb",
warden: "585add005436fea88c84b66d2f0aaa0bc4454c8165b6951989678226ebd0e27c",
shakespeare:
"e993db0e6bee1090aa64c1a7f8501e6c87c15e6eed26ce686b7efba7fcfd9541",
prober: "a34f956da1245a4f370eeedeb71e09f4dfe3d6df009b0083fc5220a30790ac59",
"qa-lead": "5beb75426761fdf952fdd889fb06c3152d90e2ead9358071d747169b10518c0e",
};

function sha256(text: string): string {
return createHash("sha256").update(text).digest("hex");
}

describe("director cards compose from prompt-fragments", () => {
test("registry system prompts are the composed cards", () => {
for (const id of DIRECTOR_IDS) {
expect(DIRECTOR_REGISTRY[id].systemPrompt).toBe(DIRECTOR_CARDS[id]);
}
});

test("composed output hash pin", () => {
for (const id of DIRECTOR_IDS) {
expect(sha256(DIRECTOR_CARDS[id])).toBe(CARD_SHA256[id]);
}
});

test("shared frames are the fragment helpers, not a second wording", () => {
expect(DIRECTOR_CARDS.explorer.startsWith(identityLine("Explorer"))).toBe(
true,
Expand All @@ -72,17 +40,4 @@ describe("director cards compose from prompt-fragments", () => {
DIRECTOR_IDS.filter((id) => id !== "dispatch").sort(),
);
});

test("director modules do not embed a second copy of the card", () => {
for (const id of DIRECTOR_IDS) {
const path =
id === "qa-lead"
? new URL("./qa-lead/package.ts", import.meta.url)
: new URL(`../../../agents/${id}/src/prompt.ts`, import.meta.url);
const text = readFileSync(path, "utf8");
expect(text).not.toContain("PRIMARY INTENT");
expect(text).not.toContain("OUT OF LANE:");
expect(text).not.toContain("# Route");
}
});
});
137 changes: 0 additions & 137 deletions src/agent/model-family-policy.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -190,143 +190,6 @@ describe("resolveModelFamilyPolicy", () => {
expect(gpt.advertisedToolDeny).toEqual([]);
});

describe("astra repro trace (CL-9027)", () => {
// Minimal failing session-trace fixture: 8 consecutive tool-only turns
// from a gpt-6-astra leaf. The shared guard fires only on exact repeats,
// so a loop varying trivial argument details escapes it — evasion, not
// waste — and the residual forbids exactly this variation. Test-only
// forensics; no signature normalization ships in first-party code.
interface ReproTurn {
tool: string;
args: Record<string, unknown>;
resultTokens: number;
}
const ASTRA_REPRO_TRACE: readonly ReproTurn[] = [
{
tool: "read",
args: { path: "src/agent/prompts.ts" },
resultTokens: 3200,
},
{
tool: "read",
args: { path: "./src/agent/prompts.ts" },
resultTokens: 3200,
},
{
tool: "read",
args: { path: "src/agent/prompts.ts", offset: 1 },
resultTokens: 3180,
},
{
tool: "grep",
args: { pattern: "narrate", path: "src/agent" },
resultTokens: 420,
},
{
tool: "read",
args: { path: "src/agent/prompts.ts" },
resultTokens: 3200,
},
{
tool: "grep",
args: { pattern: "narrate ", path: "src/agent" },
resultTokens: 420,
},
{
tool: "read",
args: { path: "./src/agent/prompts.ts", limit: 2000 },
resultTokens: 3200,
},
{
tool: "read",
args: { path: "src/agent/prompts.ts", offset: 0 },
resultTokens: 3200,
},
];

function stableJson(value: unknown): string {
if (value === null || typeof value !== "object")
return JSON.stringify(value);
if (Array.isArray(value)) return `[${value.map(stableJson).join(",")}]`;
const obj = value as Record<string, unknown>;
const keys = Object.keys(obj).sort();
return `{${keys.map((k) => `${JSON.stringify(k)}:${stableJson(obj[k])}`).join(",")}}`;
}

function fingerprint(turn: ReproTurn): string {
return `${turn.tool}:${stableJson(turn.args)}`;
}

function maxExactRepeats(fps: readonly string[]): number {
let best = 1;
for (let period = 1; period <= 6; period++) {
for (let end = 1; end <= fps.length; end++) {
const prefix = fps.slice(0, end);
let i = prefix.length - 1;
let j = i - period;
let matched = 0;
while (j >= 0 && prefix[i] === prefix[j]) {
matched++;
i--;
j--;
}
const reps = Math.floor((matched + period) / period);
if (reps > best) best = reps;
}
}
return best;
}

function evasionKey(turn: ReproTurn): string {
const args = { ...turn.args };
// The trivial deltas this trace varies: a leading ./ prefix, default
// offset/limit values, and padding whitespace.
if (typeof args.path === "string")
args.path = args.path.replace(/^\.\//, "");
if (args.offset === 0 || args.offset === 1) delete args.offset;
if (typeof args.limit === "number") delete args.limit;
if (typeof args.pattern === "string") args.pattern = args.pattern.trim();
return `${turn.tool}:${stableJson(args)}`;
}

function classifyAstraTrace(trace: readonly ReproTurn[]): string {
if (maxExactRepeats(trace.map(fingerprint)) >= 3) return "waste";
const groups = new Map<string, number>();
for (const turn of trace) {
const key = evasionKey(turn);
groups.set(key, (groups.get(key) ?? 0) + 1);
}
return Math.max(...groups.values()) >= 3 ? "evasion" : "waste";
}

test("classifies as evasion: no exact repeat trips the shared guard", () => {
expect(ASTRA_REPRO_TRACE).toHaveLength(8);
expect(maxExactRepeats(ASTRA_REPRO_TRACE.map(fingerprint))).toBeLessThan(
3,
);
expect(classifyAstraTrace(ASTRA_REPRO_TRACE)).toBe("evasion");
});

test("pins the offending pattern and the wasted turn/token delta", () => {
const groups = new Map<string, number>();
for (const turn of ASTRA_REPRO_TRACE) {
const key = evasionKey(turn);
groups.set(key, (groups.get(key) ?? 0) + 1);
}
// Six re-reads of one file plus two re-greps of one pattern, each run
// differing only by a trivial argument delta.
expect(groups.get('read:{"path":"src/agent/prompts.ts"}')).toBe(6);
expect(groups.get('grep:{"path":"src/agent","pattern":"narrate"}')).toBe(
2,
);
const wastedTokens = ASTRA_REPRO_TRACE.reduce(
(sum, turn) => sum + turn.resultTokens,
0,
);
expect(wastedTokens).toBe(20020);
});
});

test("astra resolves to astra with the composed residual; thresholds stay default (CL-9027)", () => {
const base = resolveModelFamilyPolicy({
providerName: "unknown-provider",
Expand Down
Loading
Loading