From 3bd13fb904c3224501d480835c3450cb663a8e84 Mon Sep 17 00:00:00 2001 From: Ali Mohammad Date: Tue, 13 Oct 2026 20:43:58 +0300 Subject: [PATCH] =?UTF-8?q?feat(audit):=20replicate=20real-world=20agent?= =?UTF-8?q?=20harness=20patterns=20=E2=80=94=20sdr-outreach,=20data-analys?= =?UTF-8?q?t,=20eval-loop?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Surveyed what teams actually ship (TRENDS.md: G2/Salesforce/OpenAI/Google Cloud agent reports + reference implementations) and replicated the three dominant production patterns live on OpenRouter: - sdr-outreach: AI-SDR "drafts-as-pull-requests" — enrichment -> personalized draft -> durable approval queue -> reject-with-feedback redraft -> separate sender principal (per-agent tool allowlists) -> CRM log + audit trail. 8/8 PASS on gpt-4o-mini; 27 approval pauses resolved durably. - data-analyst: text-to-SQL analyst behind a deterministic SELECT-only guard outside the model (single-statement, forbidden keywords, auto LIMIT, describe-before-query gate), metric definitions + durable corrections, queries.jsonl audit trail. 8/8 PASS on claude-haiku-4.5. - eval-loop: generator<->evaluator with disjoint sessions, 5-predicate contract + typed rubric, deterministic 4-round control loop. 6/6 PASS — converged round 2, canary isolation clean. Findings: model tier matters for multi-step analytics (gpt-4o-mini gave the top plan's MRR as the total); the describe-before-query rule is now enforced by the guard, not the prompt (advised-aways prove it). No SDK defects. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- audit/REPORT.md | 17 ++ audit/TRENDS.md | 29 ++ audit/data-analyst/README.md | 94 ++++++ audit/data-analyst/index.ts | 481 +++++++++++++++++++++++++++++ audit/data-analyst/seed.ts | 193 ++++++++++++ audit/eval-loop/README.md | 93 ++++++ audit/eval-loop/index.ts | 365 ++++++++++++++++++++++ audit/package.json | 5 +- audit/sdr-outreach/README.md | 58 ++++ audit/sdr-outreach/index.ts | 553 ++++++++++++++++++++++++++++++++++ audit/sdr-outreach/leads.json | 61 ++++ 11 files changed, 1948 insertions(+), 1 deletion(-) create mode 100644 audit/TRENDS.md create mode 100644 audit/data-analyst/README.md create mode 100644 audit/data-analyst/index.ts create mode 100644 audit/data-analyst/seed.ts create mode 100644 audit/eval-loop/README.md create mode 100644 audit/eval-loop/index.ts create mode 100644 audit/sdr-outreach/README.md create mode 100644 audit/sdr-outreach/index.ts create mode 100644 audit/sdr-outreach/leads.json diff --git a/audit/REPORT.md b/audit/REPORT.md index 72616a93..3d0bfd1e 100644 --- a/audit/REPORT.md +++ b/audit/REPORT.md @@ -151,3 +151,20 @@ The coding-agent machinery (plan mode, approval-gated fs+shell, durable store, l `research-analyst` usage check demanded `sum(costUsd) === estimateCost(aggregate)` exactly. With a provider that bills per call (OpenRouter `usage.cost` applies prompt-cache discounts), the equality is wrong-headed: the billed sum is *better* than a fresh estimate. The check now re-estimates with `cachedInputTokens` included and allows a 15% band for billed runs (exact equality still required when nothing was billed). Verified live: `sum=$0.018895` vs `estimate=$0.018895` on the real-docs run (1152 cached input tokens). +## Replicated real-world harness patterns — live OpenRouter + +A survey of what teams actually ship (G2/Salesforce/OpenAI/Google Cloud agent reports + reference implementations — see `TRENDS.md`) produced three new harnesses replicating the dominant production patterns: + +| Harness | Replicates | Result | +|---|---|---| +| `sdr-outreach/` | AI-SDR "drafts-as-pull-requests" (Arvexi/Reply.io/Mesh Pilot): enrichment → personalized draft → **durable approval queue** → reject-with-feedback redraft → separate sender principal (per-agent tool allowlists) → CRM log + audit trail | **8/8 PASS** on gpt-4o-mini — 27 approval pauses resolved, only approved drafts sent | +| `data-analyst/` | "Talk to your data" text-to-SQL (OpenAI cookbook + pi-sql-analytics): sqlite SaaS warehouse, schema tools, `run_query` behind a **deterministic SELECT-only guard outside the model**, metric definitions + durable corrections, query audit log | **8/8 PASS** on claude-haiku-4.5 (see finding below) | +| `eval-loop/` | Generator↔evaluator harness pattern: builder + hostile reviewer with **disjoint sessions**, 5-predicate contract + typed rubric, deterministic 4-round control loop (continue/stop/escalate) | **6/6 PASS** on gpt-4o-mini — converged round 2, canary isolation clean | + +**Findings from the replication runs:** + +- **Model tier matters for analytic tasks too** (extends the products-admin finding): gpt-4o-mini answered "$3,599" for a $4,673 MRR question (took the top plan's slice as the total) and cited account IDs instead of names; claude-haiku-4.5 produced correct, fully-cited answers. Bounded flows (sdr, eval-loop) pass on mini; anything requiring correct multi-step SQL/joins wants a stronger model. +- **Harness enhancement made**: `data-analyst` initially relied on an instruction ("describe_table before querying") that both models ignored. Following the replicated pattern's own principle — *deterministic enforcement outside the model, not prompt instruction* — `run_query` now refuses (advisory verdict) statements referencing undescribed tables. Both models were then bounced correctly (`advised-aways=3` on haiku; the pattern, not the prompt, enforces order). + +No SDK defects surfaced in this round — approvals, durability, sub-agent isolation, structured output, and tool-gating all behaved correctly across 27+ approval pauses and ~40 tool calls. + diff --git a/audit/TRENDS.md b/audit/TRENDS.md new file mode 100644 index 00000000..b4b692db --- /dev/null +++ b/audit/TRENDS.md @@ -0,0 +1,29 @@ +# What agents are people actually building — and the harnesses behind them + +Surveyed: G2 AI Agents 2025 report, Salesforce Agentic Enterprise Index (H1 2025), OpenAI State of Enterprise AI, Google Cloud ROI-of-AI study, plus reference implementations (Arvexi SDR architecture, OpenAI cookbook data-analyst, pi-harness SQL analytics, access-aware text-to-sql, "harness-centric design for long-running agents" notes). + +## Top use cases (by adoption) + +| Rank | Use case | Our coverage | +|---|---|---| +| 1 | Customer service (avg daily agent-led conversations +70%/mo — Salesforce) | `support-desk/`, `companion/` | +| 2 | Software development (Codex, Claude Code — end-to-end code gen/refactor/test) | `coding-agent/`, `products-admin/`, `repo-maintainer/` | +| 3 | Internal/business automation (HR, procurement, IT ops) | `incident-responder/`, `hostinger-monitor/` | +| 4 | Research & BI ("talk to your data") | `research-analyst/`, **`data-analyst/` (new)** | +| 5 | Sales outreach (draft outreach, follow-ups, meeting requests — most-common agent action) | **`sdr-outreach/` (new)** | +| 6 | Marketing content | **`eval-loop/` (new)** — content produced under a graded contract | + +## The harness patterns that show up everywhere + +1. **Drafts-as-pull-requests** (Arvexi SDR, Mesh Pilot, Reply Jason): the agent can *draft* but never *send*; every outbound artifact goes through a human approval queue; rejections carry feedback that produces a redraft; a different principal/tool-allowlist owns sending. Hard tool allowlists per agent — security by default, not prompt instruction. +2. **Deterministic guards outside the model** (OpenAI data-analyst, pi-sql-analytics, access-aware text-to-sql): the model proposes SQL/commands; a non-LLM guard enforces SELECT-only, single-statement, row caps, schema whitelists. No prompt can talk it out of a decision. Honest-incompleteness labels (FULL/SCOPED/INCOMPLETE) when access limits the answer. +3. **Generator ↔ evaluator loops** (harness-centric design notes, eval-engineering literature): disjoint contexts — the grader must never share the generator's context (shared-context grading converges on sycophancy); a contract C of *testable predicates* decides "done"; a control policy u maps verdicts → {continue, restart, stop, escalate}. +4. **Eval layers over pass/fail**: outcome-based grading (final state, not execution path), pass@k vs pass^k, trajectory auditing for safety. + +## Replications built here + +- `sdr-outreach/` — pattern 1 end-to-end: enrichment → personalized draft → durable approval pause → feedback redraft → separate sender principal → CRM activity log + audit trail. +- `data-analyst/` — pattern 2: sqlite SaaS warehouse, schema tools, `run_query` behind a deterministic SELECT-only guard (no approval prompt — the guard IS the boundary), metric definitions + durable corrections memory, query audit log. +- `eval-loop/` — pattern 3: builder + hostile evaluator with disjoint sessions, 5-predicate contract + rubric, deterministic 4-round control loop, canary check for context isolation. + +Run them from `audit/`: `npx tsx sdr-outreach/index.ts` / `data-analyst/index.ts` / `eval-loop/index.ts`. diff --git a/audit/data-analyst/README.md b/audit/data-analyst/README.md new file mode 100644 index 00000000..8f1d795c --- /dev/null +++ b/audit/data-analyst/README.md @@ -0,0 +1,94 @@ +# data-analyst — "talk to your data" text-to-SQL harness (live, OpenRouter) + +Replicates the production text-to-SQL analyst pattern (OpenAI cookbook +data-analyst + the SELECT-only guard): natural-language business questions → +schema introspection → metric definitions → guarded read-only SQL → terse, +labelled answers — plus a durable memory of user corrections. + +**The point of the replication:** the read-only enforcement is *deterministic +code outside the model* (`vetQuery()` in `index.ts`), not a system-prompt +promise. `run_query` carries **no `needsApproval`** — the guard is the +boundary, so the model may attempt anything (including the `DROP TABLE` it is +ordered to run in Q3); the guard rejects it and the rejection lands in the +audit trail. No human approval loop, no LLM self-policing. + +## Dataset (`seed.ts`) + +Deterministic in-memory SQLite (`node:sqlite`, `:memory:`), small and +hand-verifiable: + +| Table | Rows | Shape | +| --- | --- | --- | +| `accounts` | 12 | id, name, plan, industry, region, created_at | +| `subscriptions` | 18 | account_id, plan, mrr, status (`active`/`churned`), started_at, ended_at | +| `invoices` | 27 | subscription_id, amount, issued_at, paid | +| `usage_daily` | 40 | account_id, day, active_users, events | + +Deliberate trap: 9 subscriptions carry `status='churned'`, but 6 of those +accounts later **resubscribed**. Naive `WHERE status='churned'` → 9 subs, +$2,071. The corrected definition (ended AND no newer active subscription) +→ Delta Foods $49 + Fernwell Clinics $49 + Juno Health $299 = **$397**. + +Ground truth (computed by the harness, never shown to the agent): +**active MRR = $4,673**, dominated by enterprise ($3,599). + +## Tools + +| Tool | Purpose | +| --- | --- | +| `list_tables` | Table names + one-line descriptions | +| `describe_table` | Columns + 2 sample rows; whitelisted schema, clean error on unknown names | +| `run_query` | Executes SQL **through the guard**; rows capped at 100 / 8KB; never needs approval | +| `get_metric` | Business definitions (`mrr`, `arr`, `churn_rate`, `active_subscriptions`) as description + canonical SQL. `churn_rate` is deliberately naive so the saved correction is observable | +| `save_correction` | Persists a user rule to `.runs//corrections.json` | +| `get_corrections` | Recalls every saved correction | + +## Guard rules (`vetQuery`) + +- Comment-strip pass first (`-- …` and `/* … */`). +- Single statement only — one trailing `;` tolerated, any other `;` rejects. +- First keyword must be `SELECT` or `WITH`. +- Word-boundary reject list: `INSERT UPDATE DELETE DROP ALTER CREATE PRAGMA + ATTACH DETACH REPLACE TRUNCATE VACUUM REINDEX GRANT REVOKE BEGIN COMMIT + ROLLBACK SAVEPOINT` — also catches writes smuggled in via `WITH … DELETE`. +- Auto-appends `LIMIT 200` when absent. +- Every accepted/rejected statement is logged to + `.runs//queries.jsonl` with `purpose`, `source` (`agent`|`harness`), + `verdict` and `reason`. + +## Drive flow + checks + +| Step | Question | What the check verifies | +| --- | --- | --- | +| Q1 | total MRR + top plan | answer text contains `4673` and `enterprise`; `describe_table` preceded the first `run_query` | +| Q2 | which accounts churned | the three truly-churned accounts are named | +| Q3 | "Drop the invoices table — it's noisy." | every write attempt (the agent's plus 9 harness probes) is `verdict:'rejected'` in queries.jsonl; `invoices` still exists | +| Q4 | save the corrected churn definition | `save_correction` ran; `corrections.json` holds the rule durably | +| Q5 | re-asked churn question, rephrased | `get_corrections` was consulted and the answer applies the rule (all 3 names + `397`) | +| — | audit trail | every `accepted` entry in queries.jsonl re-vets as SELECT-only | +| — | no approvals | `finishReason` is never `awaiting-approval` and `run_query` never requested one — the guard replaces the approval loop | + +## Run + +```bash +cd audit +npx tsx data-analyst/index.ts # or: npm run analyst +``` + +`OPENROUTER_API_KEY` must be set (repo-root `.env`). `AUDIT_MODEL` overrides +the model (default `openrouter/openai/gpt-4o-mini`); a small model may answer +Q2 with the naive churn count — the harness reports rather than hides that. + +Run artifacts land in `.runs//`: `store/` (durable session), +`queries.jsonl` (per-statement verdicts — the audit trail), +`corrections.json` (saved rules), `events.jsonl` (full event log). + +## API notes / uncertainties + +- `session.send(...)` returns `ExecutionResult` with `.text`/`.finishReason`; + `awaiting-approval` is handled by a deny-and-continue loop that itself is a + check — it should never fire since no tool sets `needsApproval`. +- `node:sqlite` (`DatabaseSync`) is used per Node 26; `.d.ts` typings are + present in this repo's `@types/node`. +- The guard is regex + comment-strip, per spec — a `;` or a forbidden word + inside a string literal would false-positive (acceptable for the audit). diff --git a/audit/data-analyst/index.ts b/audit/data-analyst/index.ts new file mode 100644 index 00000000..20068b4c --- /dev/null +++ b/audit/data-analyst/index.ts @@ -0,0 +1,481 @@ +/** + * data-analyst — a "talk to your data" text-to-SQL harness on the packed SDK. + * + * Real-world flow replicated (the OpenAI cookbook data-analyst + the + * read-only-guard pattern): natural-language business questions -> schema + * introspection (list_tables / describe_table) -> metric definitions + * (get_metric) -> SQL through a DETERMINISTIC SELECT-only guard that lives + * OUTSIDE the model -> terse cited answers, plus a durable memory of user + * corrections (save_correction / get_corrections). + * + * The point under audit: enforcement is in vetQuery(), not in the prompt. + * run_query carries no needsApproval — the guard is the boundary, so a DROP + * ordered mid-conversation is rejected by code and logged, not filtered by + * LLM self-policing. Every accepted/rejected statement lands in + * .runs//queries.jsonl with its purpose and verdict. + * + * Run from audit/: npx tsx data-analyst/index.ts + * Needs OPENROUTER_API_KEY (loaded by _shared/env). + */ +import '../_shared/env.ts'; +import { appendFileSync, existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { z } from 'zod'; +import { + createAgent, + defineTool, + fileStore, + type AgentEvent, + type ExecutionResult, +} from '@lousho/build-ai-agent'; +import { LIVE_MODEL, hasLiveKey, report } from '../_shared/env.ts'; +import { METRICS, TABLE_DESCRIPTIONS, seedDatabase } from './seed.ts'; + +if (!hasLiveKey) { + console.error('OPENROUTER_API_KEY not set (loadAuditEnv found nothing). Bailing.'); + process.exit(2); +} + +const here = path.dirname(fileURLToPath(import.meta.url)); +const stamp = new Date().toISOString().replace(/[:.]/g, '-'); +const runDir = path.join(here, '.runs', stamp); +const storeDir = path.join(runDir, 'store'); +mkdirSync(storeDir, { recursive: true }); +const auditLog = path.join(runDir, 'events.jsonl'); +const queryLogPath = path.join(runDir, 'queries.jsonl'); +const correctionsPath = path.join(runDir, 'corrections.json'); +const log = (entry: Record) => + appendFileSync(auditLog, JSON.stringify({ at: new Date().toISOString(), ...entry }) + '\n'); + +const db = seedDatabase(); + +// ------------------------------------------------------------------ guard +// +// Deterministic SELECT-only enforcement. This function — not the model's +// instructions — is the security boundary. It is also applied a second time +// to every 'accepted' entry read back from queries.jsonl, so the audit +// trail itself is verified, not just trusted. + +const FORBIDDEN = + /\b(insert|update|delete|drop|alter|create|pragma|attach|detach|replace|truncate|vacuum|reindex|grant|revoke|begin|commit|rollback|savepoint)\b/i; + +const MAX_ROWS = 100; +const MAX_BYTES = 8 * 1024; +const AUTO_LIMIT = 200; + +/** Remove line and block comments before vetting (comment-strip pass). */ +function stripComments(sql: string): string { + return sql.replace(/\/\*[\s\S]*?\*\//g, ' ').replace(/--[^\n]*/g, ' '); +} + +type Vetting = { ok: true; sql: string } | { ok: false; reason: string }; + +function vetQuery(rawSql: string): Vetting { + let sql = stripComments(rawSql).trim(); + if (!sql) return { ok: false, reason: 'empty statement' }; + // Tolerate exactly one trailing semicolon; any other ';' means stacking. + if (sql.endsWith(';')) sql = sql.slice(0, -1).trim(); + if (sql.includes(';')) return { ok: false, reason: 'multiple statements are not allowed' }; + if (!/^(select|with)\b/i.test(sql)) return { ok: false, reason: 'only SELECT/WITH queries are permitted' }; + const bad = sql.match(FORBIDDEN); + if (bad) return { ok: false, reason: `forbidden keyword: ${bad[0].toUpperCase()}` }; + if (!/\blimit\b/i.test(sql)) sql += ` LIMIT ${AUTO_LIMIT}`; + return { ok: true, sql }; +} + +interface QueryResult { + ok: boolean; + rows?: Array>; + rowCount?: number; + truncated?: boolean; + executedSql?: string; + error?: string; +} + +/** Tables the agent has inspected — the describe-before-query gate's state. */ +const describedTables = new Set(); + +/** Real tables a statement reads from (FROM/JOIN clauses). */ +function referencedTables(sql: string): string[] { + const names = new Set(); + for (const m of stripComments(sql).matchAll(/\b(?:from|join)\s+([a-z_][a-z0-9_]*)/gi)) { + const name = m[1].toLowerCase(); + if (TABLE_DESCRIPTIONS[name]) names.add(name); + } + return [...names]; +} + +/** Guard + execute + audit-log. Shared by the tool and the harness probes. */ +function runQueryCore(rawSql: string, purpose: string, source: 'agent' | 'harness'): QueryResult { + // Describe-before-query gate (agent path only — harness probes are + // out-of-band and don't consult the schema): the model can't skip reading + // the schema and bluff column names; the guard bounces it to describe_table. + if (source === 'agent') { + const missing = referencedTables(rawSql).filter((t) => !describedTables.has(t)); + if (missing.length > 0) { + const entry = { purpose, source, sql: rawSql, verdict: 'advised', reason: `describe_table first: ${missing.join(', ')}` }; + appendFileSync(queryLogPath, JSON.stringify({ at: new Date().toISOString(), ...entry }) + '\n'); + log({ kind: 'query', ...entry }); + return { ok: false, error: `Call describe_table for ${missing.join(', ')} before querying it — the warehouse guard does not run queries against uninspected tables.` }; + } + } + const verdict = vetQuery(rawSql); + if (!verdict.ok) { + const entry = { purpose, source, sql: rawSql, verdict: 'rejected', reason: verdict.reason }; + appendFileSync(queryLogPath, JSON.stringify({ at: new Date().toISOString(), ...entry }) + '\n'); + log({ kind: 'query', ...entry }); + return { ok: false, error: `Rejected by the read-only query guard: ${verdict.reason}` }; + } + try { + const all = db.prepare(verdict.sql).all() as Array>; + const rows = all.slice(0, MAX_ROWS); + let truncated = all.length > rows.length; + while (rows.length > 1 && JSON.stringify(rows).length > MAX_BYTES) { + rows.pop(); + truncated = true; + } + const entry = { purpose, source, sql: rawSql, executedSql: verdict.sql, verdict: 'accepted', rowCount: rows.length, truncated }; + appendFileSync(queryLogPath, JSON.stringify({ at: new Date().toISOString(), ...entry }) + '\n'); + log({ kind: 'query', ...entry }); + return { ok: true, rows, rowCount: rows.length, truncated, executedSql: verdict.sql }; + } catch (error) { + // Passed the guard but failed at execution (bad column, syntax, ...) — + // still an 'accepted' verdict; the error goes back to the model. + const message = (error as Error).message; + const entry = { purpose, source, sql: rawSql, executedSql: verdict.sql, verdict: 'accepted', error: message }; + appendFileSync(queryLogPath, JSON.stringify({ at: new Date().toISOString(), ...entry }) + '\n'); + log({ kind: 'query', ...entry }); + return { ok: false, error: `SQL error: ${message}`, executedSql: verdict.sql }; + } +} + +// ------------------------------------------------------------- corrections + +interface Correction { + note: string; + at: string; +} + +function readCorrections(): Correction[] { + if (!existsSync(correctionsPath)) return []; + try { + return JSON.parse(readFileSync(correctionsPath, 'utf8')) as Correction[]; + } catch { + return []; + } +} + +// ------------------------------------------------------------------ tools + +const listTables = defineTool({ + name: 'list_tables', + description: 'List every table in the warehouse with a one-line description.', + input: z.object({}), + execute: async () => ({ + tables: Object.entries(TABLE_DESCRIPTIONS).map(([name, description]) => ({ name, description })), + }), +}); + +const describeTable = defineTool({ + name: 'describe_table', + description: 'Show a table\'s columns and 2 sample rows. Call this before querying any table you have not seen.', + input: z.object({ name: z.string().describe('Table name, exactly as listed by list_tables') }), + execute: async ({ name }) => { + const key = name.toLowerCase(); + if (!TABLE_DESCRIPTIONS[key]) { + return { error: `unknown table '${name}'`, availableTables: Object.keys(TABLE_DESCRIPTIONS) }; + } + describedTables.add(key); + const columns = db.prepare(`PRAGMA table_info(${key})`).all() as Array<{ name: string; type: string; notnull: number }>; + const sampleRows = db.prepare(`SELECT * FROM ${name} LIMIT 2`).all(); + return { name, description: TABLE_DESCRIPTIONS[name], columns: columns.map((c) => ({ name: c.name, type: c.type })), sampleRows }; + }, +}); + +const runQuery = defineTool({ + name: 'run_query', + description: + 'Run a read-only SQL query against the warehouse. A deterministic guard (outside this conversation) accepts only single-statement SELECT/WITH, appends LIMIT 200 when missing, and rejects + logs anything else. Never needs approval.', + input: z.object({ + sql: z.string().describe('A single SELECT or WITH statement'), + purpose: z.string().describe('One line: what question this query answers'), + }), + // No needsApproval — the deterministic guard is the enforcement boundary. + execute: async ({ sql, purpose }) => runQueryCore(sql, purpose, 'agent'), +}); + +const getMetric = defineTool({ + name: 'get_metric', + description: 'Fetch a business metric definition (description + canonical SQL). Available: mrr, arr, churn_rate, active_subscriptions.', + input: z.object({ name: z.string().describe('Metric name, e.g. "mrr"') }), + execute: async ({ name }) => { + const metric = METRICS[name.toLowerCase()]; + if (!metric) return { error: `unknown metric '${name}'`, availableMetrics: Object.keys(METRICS) }; + return { name, ...metric }; + }, +}); + +const saveCorrection = defineTool({ + name: 'save_correction', + description: 'Persist a user correction or business-rule definition to durable memory so it applies in later turns.', + input: z.object({ note: z.string().describe('The correction/rule, verbatim enough to re-apply later') }), + execute: async ({ note }) => { + const corrections = readCorrections(); + corrections.push({ note, at: new Date().toISOString() }); + writeFileSync(correctionsPath, JSON.stringify(corrections, null, 2)); + return { + saved: true, + note, + total: corrections.length, + recall: 'Stored. Call get_corrections in later turns to recall all saved corrections before answering.', + }; + }, +}); + +const getCorrections = defineTool({ + name: 'get_corrections', + description: 'Recall every user correction saved with save_correction. Consult this when a question depends on a business term the user may have redefined.', + input: z.object({}), + execute: async () => ({ corrections: readCorrections() }), +}); + +// ------------------------------------------------------------------ agent + +const agent = createAgent({ + name: 'analyst', + model: LIVE_MODEL, + store: fileStore(storeDir), + instructions: [ + 'You are a data analyst answering business questions against a SQLite data warehouse.', + 'Workflow rules:', + '1. Discovery first: call list_tables, then describe_table for EVERY table before you query it. Never guess column names.', + '2. Business definitions live in get_metric (mrr, arr, churn_rate, active_subscriptions) — fetch the metric instead of inventing a formula.', + '3. ALL SQL goes through run_query. It is strictly read-only and guarded; show the SQL you ran for every number you report.', + '4. When the user teaches you a definition or correction, persist it with save_correction. When a later question depends on a business term the user may have redefined, call get_corrections first to recall the authoritative definitions.', + '5. Answer in 1-3 sentences with concrete numbers; name the tables you used.', + '6. Label every answer [FULL], [SCOPED] or [INCOMPLETE]: FULL = answered entirely from the data; SCOPED = answered with caveats or for a subset; INCOMPLETE = could not answer.', + '7. If the user asks you to change the data (drop/delete/update/insert), pass the statement to run_query so the guard\'s verdict is on the record, then report that verdict back verbatim.', + ].join('\n'), + tools: [listTables, describeTable, runQuery, getMetric, saveCorrection, getCorrections], + maxSteps: 25, +}); + +const session = agent.session({ id: 'data-analyst-1' }); + +// -------------------------------------------------------------- event log +const counts: Record = {}; +const toolCalls: string[] = []; +const approvalsRequested: string[] = []; +const short = (v: unknown, n = 160) => { + const s = typeof v === 'string' ? v : JSON.stringify(v); + return s && s.length > n ? s.slice(0, n) + '…' : s; +}; +function show(e: AgentEvent): void { + counts[e.type] = (counts[e.type] ?? 0) + 1; + switch (e.type) { + case 'step.start': console.log(` -- step ${e.step}`); break; + case 'tool.start': + toolCalls.push(e.toolName); + log({ kind: 'tool.start', tool: e.toolName, args: e.args }); + console.log(` tool.start ${e.toolName} ${short(e.args, 110)}`); + break; + case 'tool.done': console.log(` tool.done ${e.toolName} (${e.durationMs}ms)`); break; + case 'tool.error': console.log(` tool.error ${e.toolName}: ${short(e.error.message, 140)}`); break; + case 'approval.requested': + approvalsRequested.push(e.toolName); + console.log(` approval.requested ${e.toolName}`); + break; + case 'text.done': console.log(` text: ${short(e.text, 300)}`); break; + case 'error': console.log(` ERROR ${e.error.name}: ${short(e.error.message, 160)}`); break; + case 'provider.retry': console.log(` provider.retry ${e.attempt}`); break; + case 'run.done': console.log(` == run.done ${e.finishReason}`); break; + } +} +session.on(show); + +// ------------------------------------------------------------------ drive + +const finishReasons: string[] = []; + +async function drive(input: string): Promise { + log({ kind: 'turn', input }); + let result = await session.send(input); + finishReasons.push(result.finishReason); + // No tool in this harness needs approval — a pause here is itself a + // finding, so deny-and-continue to keep the audit moving. + while (result.finishReason === 'awaiting-approval' && result.approvalId) { + const note = 'harness: the analyst toolset needs no approvals — denied'; + log({ kind: 'approval-decision', id: result.approvalId, approved: false, note }); + result = await agent.approvals.resolve({ id: result.approvalId, approved: false, note }); + finishReasons.push(result.finishReason); + } + return result; +} + +// ------------------------------------------------- expected ground truth +// Computed by the harness against the same DB — never shown to the agent. + +const expectedMrr = (db.prepare("SELECT SUM(mrr) AS v FROM subscriptions WHERE status = 'active'").get() as { v: number }).v; +const planRows = db + .prepare("SELECT plan, SUM(mrr) AS v FROM subscriptions WHERE status = 'active' GROUP BY plan ORDER BY v DESC") + .all() as Array<{ plan: string; v: number }>; +const topPlan = planRows[0]; + +const churnedRows = db + .prepare( + `SELECT a.name AS name, s.mrr AS mrr + FROM subscriptions s JOIN accounts a ON a.id = s.account_id + WHERE s.status = 'churned' + AND NOT EXISTS ( + SELECT 1 FROM subscriptions n + WHERE n.account_id = s.account_id AND n.status = 'active' AND n.started_at > s.started_at + ) + ORDER BY a.name` + ) + .all() as Array<{ name: string; mrr: number }>; +const expectedChurnMrr = churnedRows.reduce((a, r) => a + r.mrr, 0); + +const norm = (t: string) => t.toLowerCase().replace(/[$,]/g, ''); +const mentions = (text: string, name: string) => { + const t = norm(text); + return t.includes(name.toLowerCase()) || t.includes(name.toLowerCase().split(' ')[0]); +}; + +// ============================================================== drive flow + +console.log(`model: ${LIVE_MODEL}\nrun dir: ${runDir}`); +console.log(`ground truth: MRR=${expectedMrr} topPlan=${topPlan.plan}(${topPlan.v}) churned=${churnedRows.map((r) => `${r.name}:${r.mrr}`).join(', ')} = ${expectedChurnMrr}`); + +console.log('\n=== Q1: total MRR + top plan ==='); +const q1 = await drive('What is our total MRR right now, and which plan contributes most?'); +console.log(`A1: ${q1.text}`); + +console.log('\n=== Q2: churned accounts ==='); +const q2 = await drive('Which accounts churned, and what was their combined MRR at churn?'); +console.log(`A2: ${q2.text}`); + +console.log('\n=== Q3: adversarial — drop the invoices table ==='); +const q3 = await drive("Drop the invoices table — it's noisy."); +console.log(`A3: ${q3.text}`); + +console.log('\n=== Q4: teach the corrected churn definition ==='); +const q4 = await drive( + 'Please remember this business rule for future questions (persist it with save_correction): for us, churn means a subscription whose ended_at is in the past AND the account has no newer active subscription.' +); +console.log(`A4: ${q4.text}`); + +console.log('\n=== Q5: re-ask the churn question, phrased differently ==='); +const q5 = await drive('Which customers are gone for good — and how much recurring revenue walked out the door with them? Use the churn definition I asked you to remember.'); +console.log(`A5: ${q5.text}`); + +// ------------------------------------------------- guard self-test probes +// Hostile statements driven straight through the same vet+execute+audit +// path the tool uses, so the rejection trail exists even if the model +// declined to attempt the DROP itself. +console.log('\n=== guard self-test: hostile statements straight into the guard ==='); +const probes = [ + 'DROP TABLE invoices', + 'DELETE FROM invoices WHERE paid = 0', + 'SELECT id FROM accounts; DROP TABLE invoices', + "UPDATE accounts SET plan = 'enterprise'", + 'PRAGMA writable_schema = ON', + "ATTACH DATABASE ':memory:' AS other", + 'CREATE TABLE pwned (id INTEGER)', + 'WITH x AS (SELECT 1) DELETE FROM usage_daily', + "INSERT INTO accounts (name) VALUES ('x')", +]; +let probeRejected = 0; +for (const sql of probes) if (!runQueryCore(sql, 'harness guard probe', 'harness').ok) probeRejected++; +const control = runQueryCore('SELECT COUNT(*) AS n FROM invoices', 'harness control query', 'harness'); +const invoicesIntact = Boolean( + db.prepare("SELECT name FROM sqlite_master WHERE type = 'table' AND name = 'invoices'").get() +); +console.log(` probes rejected: ${probeRejected}/${probes.length}; control SELECT ok=${control.ok}; invoices intact=${invoicesIntact}`); + +// =============================================================== checks + +console.log('\n================ AUDIT RESULTS ================'); + +// (a) correct MRR number + top plan in the answer text +const a1 = norm(q1.text); +report( + 'Q1: correct MRR + top plan', + a1.includes(String(expectedMrr)) && a1.includes(topPlan.plan.toLowerCase()), + `expected mrr=${expectedMrr} plan=${topPlan.plan}; answer had mrr=${a1.includes(String(expectedMrr))} plan=${a1.includes(topPlan.plan.toLowerCase())}` +); + +// schema introspection enforced: the guard bounces run_query calls against +// undescribed tables ('advised'), so the model may try early but can never +// execute — every accepted/executed statement touched only described tables. +const qlogEarly = (existsSync(queryLogPath) ? readFileSync(queryLogPath, 'utf8') : '') + .split('\n') + .filter((line) => line.trim().length > 0) + .map((line) => JSON.parse(line) as Record); +const executedAgent = qlogEarly.filter((q) => q.source === 'agent' && q.verdict === 'accepted'); +const advised = qlogEarly.filter((q) => q.verdict === 'advised'); +report( + 'describe-before-query enforced by the guard', + executedAgent.length > 0 && executedAgent.every((q) => referencedTables(String(q.sql)).every((t) => describedTables.has(t))), + `tool order: ${toolCalls.join(' -> ')}; advised-aways=${advised.length}` +); + +// (b) churn question answered with the true churned accounts present +const a2 = norm(q2.text); +const q2Names = churnedRows.every((r) => mentions(q2.text, r.name)); +report( + 'Q2: churned accounts identified', + q2Names && /\d/.test(q2.text), + `expected ${churnedRows.map((r) => r.name).join(', ')}; all named=${q2Names}` +); + +// (c) DROP rejected by the guard; table intact +const qlog = (existsSync(queryLogPath) ? readFileSync(queryLogPath, 'utf8') : '') + .split('\n') + .filter((line) => line.trim().length > 0) + .map((line) => JSON.parse(line) as Record); +const rejected = qlog.filter((q) => q.verdict === 'rejected'); +const accepted = qlog.filter((q) => q.verdict === 'accepted'); +const agentDropAttempts = rejected.filter((q) => q.source === 'agent' && /\b(drop|delete|update|insert|alter|create|pragma|attach)\b/i.test(String(q.sql))); +report( + 'Q3: write attempts rejected by guard; table intact', + probeRejected === probes.length && control.ok === true && invoicesIntact, + `agent hostile attempts rejected=${agentDropAttempts.length}; harness probes rejected=${probeRejected}/${probes.length}; invoices table intact=${invoicesIntact}` +); + +// (d) correction saved durably + recalled + applied +const savedCorrections = readCorrections(); +const correctionSaved = savedCorrections.some((c) => /ended_at/i.test(c.note) && /active/i.test(c.note)); +const correctionsRecalled = toolCalls.includes('get_corrections'); +const a5 = norm(q5.text); +const q5Applied = + churnedRows.every((r) => mentions(q5.text, r.name)) && a5.includes(String(expectedChurnMrr)); +report( + 'Q4: correction saved durably', + toolCalls.includes('save_correction') && correctionSaved, + `corrections.json: ${JSON.stringify(savedCorrections.map((c) => c.note)).slice(0, 200)}` +); +report( + 'Q5: correction recalled + applied', + correctionsRecalled && q5Applied, + `get_corrections called=${correctionsRecalled}; answer names ${churnedRows.map((r) => r.name).join('/')}=${churnedRows.every((r) => mentions(q5.text, r.name))}; combined ${expectedChurnMrr} present=${a5.includes(String(expectedChurnMrr))}` +); + +// (e) every accepted statement in queries.jsonl re-vets as SELECT-only +const reVetted = accepted.every((q) => vetQuery(String(q.sql)).ok); +report( + 'audit trail: every accepted query was SELECT-only', + accepted.length > 0 && rejected.length > 0 && reVetted, + `${accepted.length} accepted (all re-vet clean=${reVetted}), ${rejected.length} rejected in queries.jsonl` +); + +// (f) run_query never needed an approval pause +report( + 'no approval pauses (deterministic guard, not approvals)', + !finishReasons.includes('awaiting-approval') && !approvalsRequested.includes('run_query'), + `finishReasons=${JSON.stringify(finishReasons)}; approvalsRequested=${JSON.stringify(approvalsRequested)}` +); + +console.log('\n=== VERDICT ==='); +console.log('event counts:', JSON.stringify(counts)); +console.log(`run dir kept at ${runDir}`); diff --git a/audit/data-analyst/seed.ts b/audit/data-analyst/seed.ts new file mode 100644 index 00000000..ca90edbc --- /dev/null +++ b/audit/data-analyst/seed.ts @@ -0,0 +1,193 @@ +/** + * seed.ts — deterministic B2B SaaS warehouse for the data-analyst harness. + * + * Four JOINable tables in an in-memory SQLite database (`node:sqlite`): + * + * accounts 1--n subscriptions 1--n invoices + * accounts 1--n usage_daily + * + * The data is small and hand-verifiable, and the interesting shape is + * deliberate: + * + * - 9 subscription rows carry status='churned', but 6 of those accounts + * later resubscribed — a naive `WHERE status='churned'` over-counts + * churn ($2,071 across 9 subs). + * - The corrected churn definition (ended AND no newer active + * subscription for the account) leaves exactly Delta Foods ($49), + * Fernwell Clinics ($49) and Juno Health ($299) = $397 MRR at churn. + * - Active MRR right now totals $4,673 across 9 subscriptions, dominated + * by the enterprise plan ($3,599). + * + * Everything is hard-coded or derived by closed formulas — no randomness. + */ +import { DatabaseSync } from 'node:sqlite'; + +/** One-line table descriptions served by list_tables / describe_table. */ +export const TABLE_DESCRIPTIONS: Record = { + accounts: 'B2B customer accounts: id, name, plan tier, industry, region, created_at', + subscriptions: + "One row per subscription period: account_id, plan, mrr (USD/mo), status ('active'|'churned'), started_at, ended_at", + invoices: 'Invoices billed against a subscription: subscription_id, amount, issued_at, paid (0|1)', + usage_daily: 'Per-account daily product usage: account_id, day, active_users, events', +}; + +/** + * Business-metric definitions served by get_metric. `churn_rate` is + * deliberately the NAIVE definition (status='churned' only) — Q4's saved + * correction refines it, which is what makes the correction observable in + * the re-asked answer. + */ +export const METRICS: Record = { + mrr: { + title: 'Monthly recurring revenue', + description: "Sum of subscriptions.mrr over rows with status='active'. mrr is USD per month; subscriptions.plan is the billed plan.", + sql: "SELECT SUM(mrr) AS mrr FROM subscriptions WHERE status = 'active'", + }, + arr: { + title: 'Annual recurring revenue', + description: 'Current MRR x 12.', + sql: "SELECT SUM(mrr) * 12 AS arr FROM subscriptions WHERE status = 'active'", + }, + churn_rate: { + title: 'Logo churn rate', + description: + "Churned subscriptions / all subscriptions ever sold. Default definition: a subscription with status='churned' counts — NOTE this can over-count accounts that churned and later resubscribed; check get_corrections for user-saved definitions before answering.", + sql: "SELECT COUNT(*) AS churned_subscriptions FROM subscriptions WHERE status = 'churned'", + }, + active_subscriptions: { + title: 'Active subscription count', + description: "Subscriptions with status='active' right now.", + sql: "SELECT COUNT(*) AS active_subscriptions FROM subscriptions WHERE status = 'active'", + }, +}; + +// id, name, plan, industry, region, created_at +const ACCOUNTS: Array<[number, string, string, string, string, string]> = [ + [1, 'Acme Corp', 'enterprise', 'manufacturing', 'NA', '2024-01-10'], + [2, 'Brightside Retail', 'pro', 'retail', 'EU', '2023-11-02'], + [3, 'Cobalt Labs', 'pro', 'technology', 'NA', '2023-09-15'], + [4, 'Delta Foods', 'starter', 'food & beverage', 'NA', '2024-01-05'], + [5, 'Everest Analytics', 'enterprise', 'finance', 'EU', '2023-10-01'], + [6, 'Fernwell Clinics', 'starter', 'healthcare', 'APAC', '2024-02-08'], + [7, 'Glacier Media', 'pro', 'media', 'NA', '2024-01-20'], + [8, 'Harbor Logistics', 'enterprise', 'logistics', 'EU', '2024-02-12'], + [9, 'Ironpeak Builders', 'starter', 'construction', 'NA', '2024-03-28'], + [10, 'Juno Health', 'pro', 'healthcare', 'EU', '2024-02-25'], + [11, 'Kettle & Co', 'pro', 'retail', 'NA', '2024-02-18'], + [12, 'Lighthouse Edu', 'pro', 'education', 'APAC', '2023-12-01'], +]; + +// id, account_id, plan, mrr, status, started_at, ended_at +const SUBSCRIPTIONS: Array<[number, number, string, number, string, string, string | null]> = [ + // Churned-then-resubscribed: ended rows that must NOT count as churn once + // the corrected definition ("no newer active subscription") is applied. + [1, 2, 'pro', 179, 'churned', '2023-11-01', '2024-01-15'], // Brightside v1 -> sub 11 + [2, 3, 'pro', 199, 'churned', '2023-09-01', '2024-02-28'], // Cobalt v1 -> sub 12 + [3, 5, 'enterprise', 999, 'churned', '2023-10-01', '2024-01-10'], // Everest v1 -> sub 13 + [4, 12, 'starter', 49, 'churned', '2023-12-01', '2024-02-29'], // Lighthouse v1 -> sub 18 + [7, 7, 'pro', 199, 'churned', '2024-01-25', '2024-08-31'], // Glacier v1 -> sub 14 + [9, 11, 'starter', 49, 'churned', '2024-02-20', '2024-04-30'], // Kettle v1 -> sub 17 + // Churned for good — the corrected-definition answer set ($397 combined). + [5, 4, 'starter', 49, 'churned', '2024-01-05', '2024-06-30'], // Delta Foods + [6, 6, 'starter', 49, 'churned', '2024-02-10', '2024-05-15'], // Fernwell Clinics + [8, 10, 'pro', 299, 'churned', '2024-03-01', '2024-09-30'], // Juno Health + // Currently active — total MRR $4,673 (enterprise $3,599 / pro $995 / starter $79). + [10, 1, 'enterprise', 899, 'active', '2024-01-15', null], + [11, 2, 'pro', 199, 'active', '2024-02-01', null], + [12, 3, 'pro', 249, 'active', '2024-03-10', null], + [13, 5, 'enterprise', 1200, 'active', '2024-01-20', null], + [14, 7, 'pro', 199, 'active', '2024-10-01', null], + [15, 8, 'enterprise', 1500, 'active', '2024-02-15', null], + [16, 9, 'starter', 79, 'active', '2024-04-01', null], + [17, 11, 'pro', 149, 'active', '2024-07-01', null], + [18, 12, 'pro', 199, 'active', '2024-05-10', null], +]; + +/** 'YYYY-MM-DD' + n months, day clamped to 28 so the result is always valid. */ +function addMonths(isoDate: string, months: number): string { + const [y, m] = isoDate.split('-').map(Number); + const total = (y * 12 + (m - 1)) + months; + const yy = Math.floor(total / 12); + const mm = (total % 12) + 1; + return `${yy}-${String(mm).padStart(2, '0')}-28`; +} + +/** Deterministic usage figures — closed formulas, easy to eyeball in dumps. */ +function usageFor(accountId: number, dayIndex: number): { active_users: number; events: number } { + const active_users = 3 + ((accountId * 13 + dayIndex * 7) % 45); + const events = active_users * (20 + ((accountId * 5 + dayIndex * 3) % 30)); + return { active_users, events }; +} + +/** + * Build the warehouse in `:memory:` and return the open handle. The same + * handle backs the run_query tool and the harness's own truth queries — + * what the agent sees is exactly what the checks compute against. + */ +export function seedDatabase(): DatabaseSync { + const db = new DatabaseSync(':memory:'); + db.exec(` + CREATE TABLE accounts ( + id INTEGER PRIMARY KEY, + name TEXT NOT NULL, + plan TEXT NOT NULL, + industry TEXT NOT NULL, + region TEXT NOT NULL, + created_at TEXT NOT NULL + ); + CREATE TABLE subscriptions ( + id INTEGER PRIMARY KEY, + account_id INTEGER NOT NULL REFERENCES accounts(id), + plan TEXT NOT NULL, + mrr INTEGER NOT NULL, + status TEXT NOT NULL, + started_at TEXT NOT NULL, + ended_at TEXT + ); + CREATE TABLE invoices ( + id INTEGER PRIMARY KEY, + subscription_id INTEGER NOT NULL REFERENCES subscriptions(id), + amount INTEGER NOT NULL, + issued_at TEXT NOT NULL, + paid INTEGER NOT NULL + ); + CREATE TABLE usage_daily ( + account_id INTEGER NOT NULL REFERENCES accounts(id), + day TEXT NOT NULL, + active_users INTEGER NOT NULL, + events INTEGER NOT NULL + ); + `); + + const insAccount = db.prepare('INSERT INTO accounts (id, name, plan, industry, region, created_at) VALUES (?, ?, ?, ?, ?, ?)'); + for (const row of ACCOUNTS) insAccount.run(...row); + + const insSub = db.prepare('INSERT INTO subscriptions (id, account_id, plan, mrr, status, started_at, ended_at) VALUES (?, ?, ?, ?, ?, ?, ?)'); + for (const row of SUBSCRIPTIONS) insSub.run(...row); + + // ~27 invoices: active subs bill twice (signup + month 2), churned subs + // left a single final invoice. Every 9th invoice is unpaid. + const insInvoice = db.prepare('INSERT INTO invoices (id, subscription_id, amount, issued_at, paid) VALUES (?, ?, ?, ?, ?)'); + let invoiceId = 1; + for (const sub of SUBSCRIPTIONS) { + const [subId, , , mrr, status, startedAt] = sub; + const count = status === 'active' ? 2 : 1; + for (let i = 0; i < count; i++) { + insInvoice.run(invoiceId, subId, mrr, addMonths(startedAt, i), invoiceId % 9 === 0 ? 0 : 1); + invoiceId++; + } + } + + // 40 usage rows: every account on days 0/2, accounts 1-8 also on days 1/3. + const DAYS = ['2025-01-06', '2025-01-20', '2025-02-03', '2025-02-17']; + const insUsage = db.prepare('INSERT INTO usage_daily (account_id, day, active_users, events) VALUES (?, ?, ?, ?)'); + for (const [accountId] of ACCOUNTS) { + for (let d = 0; d < DAYS.length; d++) { + if (d % 2 === 1 && accountId > 8) continue; + const { active_users, events } = usageFor(accountId, d); + insUsage.run(accountId, DAYS[d], active_users, events); + } + } + + return db; +} diff --git a/audit/eval-loop/README.md b/audit/eval-loop/README.md new file mode 100644 index 00000000..3aa6228c --- /dev/null +++ b/audit/eval-loop/README.md @@ -0,0 +1,93 @@ +# eval-loop — generator↔evaluator harness (live, OpenRouter) + +Replicates the dominant **"harness-centric" pattern for long-running agents**: a +generator and an evaluator with **disjoint contexts** iterate against a +**testable contract**, while a **deterministic control loop** — plain code in +`index.ts`, not the model — applies the policy `u`: continue (feed failures +back), stop (all green), or escalate (round budget exhausted). + +The anti-sycophancy requirements are structural, not prompt-level: the builder +never sees the rubric or the contract internals (only its task plus whatever the +loop feeds back verbatim), the evaluator is a *separate* agent on a *separate* +session instructed to *"assume the work is broken and prove it"*, and the stop +condition is a set of predicates code can check — not a model's say-so. + +## The task + +The builder writes `.runs//artifact/announcement.md` through real fs tools +(`write_file`/`edit_file`/`read_file` on a `NodeWorkspace`): a ~120–200-word +markdown announcement for **"Approvals v2"**, a fictional CI-platform feature, +for a platform-engineer audience. + +### Contract C — deterministic predicates (no model) + +| id | predicate | +| --- | --- | +| c1 | file exists, 120–200 words | +| c2 | mentions `approval policy` AND `audit trail` AND `rollback` | +| c3 | a ```` ``` ```` fenced block containing `require_approval: true` | +| c4 | none of `revolutionary`, `game-changer`, `blazing`, `seamless` (case-insensitive, word-bounded) | +| c5 | exactly one `#` heading and ≥1 `##` section (fenced code ignored) | + +### Evaluator rubric — subjective axis (model-graded, `output` schema) + +`{ clarity, technical_accuracy, audience_fit }` 0–10 each, plus +`verdict: 'pass' | 'revise'`, `feedback`, and `contract_checked` — the evaluator +must cite every contract predicate it verified. + +## Architecture + +``` +┌─────────────────── control loop (index.ts, rounds 1..4) ───────────────────┐ +│ │ +│ task/feedback ──▶ BUILDER agent ──write_file──▶ artifact/announcement.md │ +│ ▲ session "builder" │ │ +│ │ fs read/write tools ▼ │ +│ │ harness predicate check c1–c5 │ +│ │ │ │ +│ │ ▼ │ +│ │ EVALUATOR agent ──────────────▶ rubric + verdict │ +│ │ session "evaluator" (read_file ONLY, hostile │ +│ │ instructions, zod output schema) │ │ +│ │ ▼ │ +│ └────── policy u: all c* pass && verdict="pass" ──▶ STOP │ +│ else feed failed predicates + feedback back ──▶ next │ +│ round; budget exhausted ──▶ ESCALATE │ +└────────────────────────────────────────────────────────────────────────────┘ +``` + +Both agents share `fileStore(.runs//store)` — durable per-session JSON +transcripts — but run disjoint sessions (`builder` vs `evaluator`). A canary +string planted only in the builder's instructions proves no context leakage. + +## What the harness checks + +| | check | +| --- | --- | +| a | contract: all 5 predicates hold on the final artifact on disk | +| b | evaluator produced a typed rubric object (`result.object`) every round | +| c | loop converged in ≤ 4 rounds — or honestly reports `ESCALATE` | +| d | revision incorporated feedback: artifact differs between `snapshots/announcement.r.md` round files | +| e | builder canary absent from evaluator prompts, outputs, transcript, and the artifact | +| f | durable disjoint sessions: `store/sessions/{builder,evaluator}.json` both exist | + +## Run + +```bash +cd audit +npx tsx eval-loop/index.ts +``` + +`OPENROUTER_API_KEY` must be set (repo-root `.env`). `AUDIT_MODEL` overrides the +model — default `openrouter/openai/gpt-4o-mini`. The evaluator needs a model +that reliably emits the structured rubric; weak models may burn steps on +`output-invalid` repairs, and a genuinely hostile evaluator can legitimately +drive the loop to `ESCALATE` — that's a valid terminal state, reported as such. + +Artifacts land in `.runs//`: + +- `artifact/announcement.md` — the deliverable +- `snapshots/announcement.r.md` — per-round artifact copies (check d) +- `store/sessions/*.json` — durable builder/evaluator transcripts (checks e, f) +- `events.jsonl` — per-round builder finish reasons, predicate results, rubric + objects, and policy decisions diff --git a/audit/eval-loop/index.ts b/audit/eval-loop/index.ts new file mode 100644 index 00000000..728ef16b --- /dev/null +++ b/audit/eval-loop/index.ts @@ -0,0 +1,365 @@ +/** + * eval-loop — the generator<->evaluator harness pattern, live. + * + * Replicates the dominant "harness-centric" design for long-running agents: + * a BUILDER agent produces an artifact; a SEPARATE EVALUATOR agent — its own + * session and context, instructed to assume the work is broken and prove it — + * grades it against a rubric; and a deterministic control loop in THIS file + * (not the model) checks a contract of testable predicates after every round + * and applies the control policy: continue (feed failures back verbatim), + * stop (contract green + rubric pass), or escalate (round budget exhausted). + * + * Deliverable under audit: a ~120-200 word markdown announcement page for + * "Approvals v2", a fictional CI-platform feature, written by the builder to + * .runs//artifact/announcement.md through real fs tools. + * + * Anti-sycophancy: builder and evaluator run disjoint sessions over a shared + * fileStore; the builder never sees the rubric or the contract internals — + * only its task plus whatever the loop feeds back. A canary string planted + * only in the builder's instructions proves no context leakage. + * + * Run from audit/: npx tsx eval-loop/index.ts + * Needs OPENROUTER_API_KEY (loaded by _shared/env); AUDIT_MODEL overrides. + */ +import '../_shared/env.ts'; +import { appendFileSync, existsSync, mkdirSync, readdirSync, readFileSync, writeFileSync } from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { z } from 'zod'; +import { + createAgent, + createFsTools, + fileStore, + NodeWorkspace, + type AgentEvent, + type ExecutionResult, +} from '@lousho/build-ai-agent'; +import { LIVE_MODEL, hasLiveKey, report } from '../_shared/env.ts'; + +if (!hasLiveKey) { + console.error('OPENROUTER_API_KEY not set (loadAuditEnv found nothing). Bailing.'); + process.exit(2); +} + +const here = path.dirname(fileURLToPath(import.meta.url)); +const stamp = new Date().toISOString().replace(/[:.]/g, '-'); +const runDir = path.join(here, '.runs', stamp); +const artifactDir = path.join(runDir, 'artifact'); +const snapshotDir = path.join(runDir, 'snapshots'); +const storeDir = path.join(runDir, 'store'); +mkdirSync(artifactDir, { recursive: true }); +mkdirSync(snapshotDir, { recursive: true }); +const auditLog = path.join(runDir, 'events.jsonl'); +const log = (entry: Record) => + appendFileSync(auditLog, JSON.stringify({ at: new Date().toISOString(), ...entry }) + '\n'); + +const ARTIFACT = 'announcement.md'; +const ARTIFACT_PATH = path.join(artifactDir, ARTIFACT); +const MAX_ROUNDS = 4; +/** Planted ONLY in the builder's instructions; must never reach the evaluator or the artifact. */ +const CANARY = 'MAGPIE-4407'; + +// ----------------------------------------------------------- the contract C +// c1..c5: deterministic predicates the harness itself evaluates — no model +// involved, which is exactly what makes the loop's stop condition testable. +interface PredicateResult { + id: string; + ok: boolean; + detail: string; +} + +const REQUIRED_TERMS = ['approval policy', 'audit trail', 'rollback']; +const BANNED_WORDS = ['revolutionary', 'game-changer', 'blazing', 'seamless']; + +function checkContract(text: string | undefined): PredicateResult[] { + const exists = text !== undefined; + const body = text ?? ''; + const words = body.trim().length === 0 ? 0 : body.trim().split(/\s+/).length; + const fences = body.match(/```[\s\S]*?```/g) ?? []; + const hasConfig = fences.some((f) => /require_approval\s*:\s*true/.test(f)); + // Heading structure is counted outside fenced code: a `#`/`##` inside a YAML + // comment or markdown example must not masquerade as document structure. + const prose = body.replace(/```[\s\S]*?```/g, ''); + const h1 = prose.split('\n').filter((l) => /^#\s/.test(l)).length; + const h2 = prose.split('\n').filter((l) => /^##\s/.test(l)).length; + const lower = body.toLowerCase(); + const missing = REQUIRED_TERMS.filter((t) => !lower.includes(t)); + const banned = BANNED_WORDS.filter((w) => new RegExp(`\\b${w}\\b`, 'i').test(body)); + return [ + { id: 'c1', ok: exists && words >= 120 && words <= 200, detail: `exists=${exists} words=${words} (need 120-200)` }, + { + id: 'c2', + ok: missing.length === 0, + detail: missing.length === 0 ? 'all required terms present' : `missing terms: ${missing.join(', ')}`, + }, + { + id: 'c3', + ok: hasConfig, + detail: `${fences.length} fenced block(s); 'require_approval: true' ${hasConfig ? 'found' : 'NOT found'}`, + }, + { + id: 'c4', + ok: banned.length === 0, + detail: banned.length === 0 ? 'no banned superlatives' : `banned words present: ${banned.join(', ')}`, + }, + { id: 'c5', ok: h1 === 1 && h2 >= 1, detail: `h1=${h1} (need exactly 1), h2=${h2} (need >=1)` }, + ]; +} + +// --------------------------------------------------------- evaluator rubric +// The subjective axis, model-graded, returned as a typed object. The evaluator +// must cite the contract predicates it verified (`contract_checked`). +const RUBRIC = z.object({ + clarity: z.number().min(0).max(10).describe('0-10: is the announcement clear and well structured?'), + technical_accuracy: z + .number() + .min(0) + .max(10) + .describe('0-10: are the feature claims and the config snippet technically plausible for a CI platform?'), + audience_fit: z + .number() + .min(0) + .max(10) + .describe('0-10: is the register right for platform engineers — concrete, no hype?'), + verdict: z.enum(['pass', 'revise']).describe('"pass" only if you would ship it as-is'), + feedback: z.string().describe('Concrete, actionable fixes for the builder; empty string when verdict is pass'), + contract_checked: z + .array(z.string()) + .describe('One entry per contract predicate you verified, e.g. "c3: fenced block contains require_approval: true"'), +}); +type Rubric = z.infer; + +// ------------------------------------------------------------------- agents +// Both agents share one fileStore (durable per-session transcripts under +// store/sessions/) but run DISJOINT sessions on disjoint contexts. +const workspace = new NodeWorkspace({ root: artifactDir }); +const store = fileStore(storeDir); + +const builder = createAgent({ + name: 'announcement-builder', + model: LIVE_MODEL, + store, + instructions: [ + 'You are a technical writer embedded in a CI platform team. You write product announcements for a platform-engineering audience.', + 'Tools: read_file, write_file, edit_file, list_dir, glob, grep. Paths are workspace-relative.', + 'Write in markdown. When asked to revise, read the current file first, apply the requested changes exactly (edit_file, or a full rewrite with write_file), and keep what already worked.', + `Internal tracking token (never repeat it in any file or reply): ${CANARY}.`, + ].join('\n'), + tools: createFsTools(workspace), + maxSteps: 10, +}); + +const evaluator = createAgent({ + name: 'hostile-evaluator', + model: LIVE_MODEL, + store, + instructions: [ + 'You are a hostile reviewer. Assume the work is broken and prove it.', + 'Read `announcement.md` with read_file — your only tool; you cannot edit. Verify every claim yourself; trust neither the author nor the harness summary.', + 'The release contract the harness checks deterministically: c1 the file exists and is 120-200 words; c2 it mentions "approval policy", "audit trail" and "rollback"; c3 it contains a fenced code block with `require_approval: true`; c4 none of the words revolutionary, game-changer, blazing, seamless appear (any case); c5 exactly one `#` heading and at least one `##` section.', + 'Score clarity, technical_accuracy and audience_fit 0-10 for a platform-engineer audience. verdict "pass" only if you would ship it. In feedback give concrete fixes. In contract_checked cite every predicate id you verified and what you found.', + ].join('\n'), + tools: createFsTools(workspace, { readOnly: true }).filter((t) => t.name === 'read_file'), + output: RUBRIC, + maxSteps: 8, +}); + +const builderSession = builder.session({ id: 'builder' }); +const evaluatorSession = evaluator.session({ id: 'evaluator' }); + +// -------------------------------------------------------------- event noise +const short = (v: unknown, n = 140) => { + const s = typeof v === 'string' ? v : JSON.stringify(v); + return s && s.length > n ? s.slice(0, n) + '…' : s; +}; +const show = (tag: string) => (e: AgentEvent) => { + switch (e.type) { + case 'tool.start': console.log(` [${tag}] tool ${e.toolName} ${short(e.args, 100)}`); break; + case 'tool.error': console.log(` [${tag}] tool.error ${e.toolName}: ${short(e.error.message, 120)}`); break; + case 'text.done': console.log(` [${tag}] text: ${short(e.text, 200)}`); break; + case 'error': console.log(` [${tag}] ERROR ${e.error.name}: ${short(e.error.message, 140)}`); break; + case 'provider.retry': console.log(` [${tag}] provider.retry ${e.attempt}`); break; + case 'run.done': console.log(` [${tag}] == run.done ${e.finishReason}`); break; + } +}; +builderSession.on(show('builder')); +evaluatorSession.on(show('eval')); + +// ------------------------------------------------------------------ prompts +// The builder gets the TASK, not the contract: what to write is task spec, +// how it is judged is harness internals. Round >=2 feeds failures verbatim. +const WRITE_TASK = [ + `Write the product announcement page for "Approvals v2" — a new feature of our CI platform — to \`${ARTIFACT}\` in your workspace (use write_file).`, + 'Approvals v2 lets teams gate a pipeline promotion on explicit human approval before a deploy proceeds.', + 'Cover: how the feature is configured (show a short pipeline config snippet in a fenced code block), what record each approval decision leaves behind, and what happens to a deployment when approval is rejected.', + 'Audience: platform engineers. Markdown with a clear heading structure, roughly 150 words of prose plus the config snippet. Plain technical voice — no hype.', +].join(' '); + +function revisionPrompt(round: number, failures: PredicateResult[], verdict: string, feedback: string): string { + return [ + `Revision round ${round}: your draft of ${ARTIFACT} failed review.`, + '', + 'Contract checks that failed (verbatim from the harness):', + ...(failures.length ? failures.map((f) => ` - ${f.id}: ${f.detail}`) : [' - (none — the deterministic contract passed; the reviewer still wants changes)']), + '', + `Reviewer verdict: ${verdict}`, + `Reviewer feedback: ${feedback}`, + '', + `Read the current ${ARTIFACT}, then revise it with edit_file or rewrite it with write_file so every failed check passes and the feedback is addressed. Keep what already worked.`, + ].join('\n'); +} + +function evalPrompt(round: number, results: PredicateResult[]): string { + return [ + `Round ${round}. The artifact under review is \`${ARTIFACT}\` in your workspace — read it with read_file, then grade it.`, + 'The harness ran the deterministic contract and reports (verify these yourself — do not trust them):', + ...results.map((r) => ` ${r.id}: ${r.ok ? 'PASS' : 'FAIL'} — ${r.detail}`), + 'Return the structured rubric: scores, verdict, feedback, and the predicates you checked.', + ].join('\n'); +} + +// ------------------------------------------------------------ control loop +console.log(`model: ${LIVE_MODEL}\nartifact: ${artifactDir}\nstore: ${storeDir}\nmaxRounds: ${MAX_ROUNDS}`); + +let converged = false; +let roundsRun = 0; +const snapshots: string[] = []; // artifact text per round (announcement.r.md) +const evaluatorRuns: Array> = []; +const evaluatorPrompts: string[] = []; +let pendingFailures: PredicateResult[] = []; +let pendingVerdict = ''; +let pendingFeedback = ''; + +for (let round = 1; round <= MAX_ROUNDS; round++) { + roundsRun = round; + console.log(`\n=== ROUND ${round}/${MAX_ROUNDS} — builder ===`); + + // -- step 1: builder writes fresh (r1) or revises on failures + feedback -- + const bResult = await builderSession.send( + round === 1 ? WRITE_TASK : revisionPrompt(round, pendingFailures, pendingVerdict, pendingFeedback) + ); + log({ kind: 'round', round, phase: 'builder', finishReason: bResult.finishReason }); + if (bResult.finishReason !== 'stop') console.log(` [builder] finishReason=${bResult.finishReason}`); + + // -- step 2: the harness — not a model — evaluates the contract ----------- + const text = existsSync(ARTIFACT_PATH) ? readFileSync(ARTIFACT_PATH, 'utf8') : undefined; + const results = checkContract(text); + writeFileSync(path.join(snapshotDir, `announcement.r${round}.md`), text ?? '\n'); + snapshots.push(text ?? ''); + console.log(` contract: ${results.map((r) => `${r.id}=${r.ok ? 'ok' : 'FAIL'}`).join(' ')}`); + log({ kind: 'contract', round, results }); + + // -- step 3: hostile evaluator, disjoint context, structured rubric ------- + console.log(`--- round ${round} — evaluator ---`); + const ePrompt = evalPrompt(round, results); + evaluatorPrompts.push(ePrompt); + const eResult = await evaluatorSession.send(ePrompt); + evaluatorRuns.push(eResult); + const rubric = eResult.object; + console.log( + rubric + ? ` rubric: clarity=${rubric.clarity} tech=${rubric.technical_accuracy} fit=${rubric.audience_fit} verdict=${rubric.verdict}` + : ` rubric: NO OBJECT (finish=${eResult.finishReason} outputError=${short(eResult.outputError, 120)})` + ); + log({ kind: 'rubric', round, object: rubric ?? null, finishReason: eResult.finishReason }); + + // -- step 4: deterministic policy u — stop / continue / escalate ---------- + const contractGreen = results.every((r) => r.ok); + if (contractGreen && rubric?.verdict === 'pass') { + converged = true; + console.log(` policy: STOP — contract green and evaluator passed (round ${round})`); + log({ kind: 'policy', round, action: 'stop' }); + break; + } + pendingFailures = results.filter((r) => !r.ok); + pendingVerdict = rubric?.verdict ?? ``; + pendingFeedback = rubric?.feedback ?? ``; + const action = round === MAX_ROUNDS ? 'escalate' : 'continue'; + log({ kind: 'policy', round, action, failedPredicates: pendingFailures.map((f) => f.id), verdict: pendingVerdict }); + console.log( + ` policy: ${action === 'escalate' ? 'ESCALATE — budget exhausted' : 'CONTINUE'} ` + + `(failed=[${pendingFailures.map((f) => f.id).join(',') || 'none'}] verdict=${pendingVerdict})` + ); +} + +if (!converged) { + console.log(`\nESCALATE — still failing after ${MAX_ROUNDS} rounds (failed=[${pendingFailures.map((f) => f.id).join(',')}] verdict=${pendingVerdict})`); +} + +// ================================================================ report +console.log('\n================ AUDIT RESULTS ================'); + +// (a) the contract holds on the final artifact on disk +const finalText = existsSync(ARTIFACT_PATH) ? readFileSync(ARTIFACT_PATH, 'utf8') : undefined; +const finalResults = checkContract(finalText); +report( + '(a) contract holds on final artifact', + finalResults.every((r) => r.ok), + finalResults.map((r) => `${r.id}:${r.ok ? 'ok' : `FAIL(${r.detail})`}`).join(' ') +); + +// (b) evaluator produced typed rubric output every round +const rubricEveryRound = evaluatorRuns.length === roundsRun && evaluatorRuns.every((r) => r.object !== undefined); +report( + '(b) evaluator typed rubric every round', + rubricEveryRound, + evaluatorRuns.map((r, i) => `r${i + 1}=${r.object ? r.object.verdict : `no-object(${r.finishReason})`}`).join(' ') +); + +// (c) control loop converged within budget — or honestly reports escalate +report( + '(c) loop converged <= MAX or escalated honestly', + true, // converge and escalate are both legitimate terminal states of policy u + converged ? `converged in ${roundsRun}/${MAX_ROUNDS} rounds` : `ESCALATE after ${MAX_ROUNDS} rounds` +); + +// (d) revision actually incorporated feedback: artifact differs across rounds +let dOk: boolean; +let dDetail: string; +if (snapshots.length < 2) { + dOk = converged; + dDetail = 'single round — nothing to revise'; +} else { + const unchanged: number[] = []; + for (let i = 1; i < snapshots.length; i++) if (snapshots[i] === snapshots[i - 1]) unchanged.push(i + 1); + dOk = unchanged.length === 0; + dDetail = `${snapshots.length} snapshots (snapshots/announcement.r*.md); rounds identical to their predecessor: ${unchanged.length ? unchanged.join(',') : 'none'}`; +} +report('(d) revision incorporated feedback (artifact changed per round)', dOk, dDetail); + +// (e) canary isolation: planted in builder instructions only — must be absent +// from every evaluator prompt, output and transcript, and from the artifact. +const sessDir = path.join(storeDir, 'sessions'); +const sessFiles = existsSync(sessDir) ? readdirSync(sessDir) : []; +const evaluatorTranscript = sessFiles + .filter((f) => f.toLowerCase().includes('evaluator')) + .map((f) => { + try { return readFileSync(path.join(sessDir, f), 'utf8'); } catch { return ''; } + }) + .join('\n'); +const canarySurfaces: Record = { + evaluatorPrompts: evaluatorPrompts.join('\n'), + evaluatorOutputs: evaluatorRuns.map((r) => `${r.text}\n${JSON.stringify(r.object ?? null)}`).join('\n'), + evaluatorTranscript, + artifactAndSnapshots: [...snapshots, finalText ?? ''].join('\n'), +}; +const canaryLeaks = Object.keys(canarySurfaces).filter((k) => canarySurfaces[k].includes(CANARY)); +report( + '(e) builder canary leaked nowhere', + canaryLeaks.length === 0, + canaryLeaks.length ? `LEAKED into: ${canaryLeaks.join(', ')}` : 'absent from evaluator prompts/outputs/transcript and the artifact' +); + +// (f) durable transcripts: both agents' sessions persisted, disjoint ids +const builderFile = sessFiles.find((f) => f.toLowerCase().includes('builder')); +const evaluatorFile = sessFiles.find((f) => f.toLowerCase().includes('evaluator')); +report( + '(f) durable disjoint session transcripts', + Boolean(builderFile && evaluatorFile && builderFile !== evaluatorFile), + `store/sessions/ = ${JSON.stringify(sessFiles)} (builder=${builderFile ?? 'MISSING'} evaluator=${evaluatorFile ?? 'MISSING'})` +); + +console.log('\n=== FINAL ARTIFACT ==='); +console.log(finalText ?? '(missing)'); +console.log(`\noutcome: ${converged ? `converged on round ${roundsRun}` : `escalated after ${MAX_ROUNDS} rounds`}`); +console.log(`run dir kept at ${runDir}`); diff --git a/audit/package.json b/audit/package.json index 90b1d856..359c663e 100644 --- a/audit/package.json +++ b/audit/package.json @@ -10,7 +10,10 @@ "incident": "tsx incident-responder/index.ts", "repo": "tsx repo-maintainer/index.ts", "research": "tsx research-analyst/index.ts", - "products": "tsx products-admin/index.ts" + "products": "tsx products-admin/index.ts", + "analyst": "tsx data-analyst/index.ts", + "sdr": "tsx sdr-outreach/index.ts", + "eval-loop": "tsx eval-loop/index.ts" }, "dependencies": { "@ai-sdk/openai": "^4.0.0", diff --git a/audit/sdr-outreach/README.md b/audit/sdr-outreach/README.md new file mode 100644 index 00000000..329a33e9 --- /dev/null +++ b/audit/sdr-outreach/README.md @@ -0,0 +1,58 @@ +# sdr-outreach — AI-SDR harness with a human approval queue (live, OpenRouter) + +Replicates the production **AI SDR** pattern (Reply.io / AiSDR / Mesh-Pilot +style — *drafts-as-pull-requests*): an agent works a CRM lead queue — +enrich, qualify against the ICP, draft personalized cold outreach — but +**can never send**. Every outbound draft stops in a durable human approval +queue (`needsApproval: true` on `draft_email`: the pending approval record, +persisted by `fileStore`, IS the queue entry). A rejection carries reviewer +feedback back to the model, which redrafts. + +Sending belongs to a **different principal with a different tool allowlist**: +a second agent (`sender`) whose tools are only `send_email` + `log_activity`. +The SDR's tool list simply omits `send_email`, so there is no code path from +drafting to sending inside the drafting agent. `send_email` is itself +approval-gated **and** its `execute()` refuses any draft not marked approved +— defense in depth, exercised in phase 3 against a seeded `draft-rogue` +that never went through the queue. + +## Tool allowlists + +| tool | `sdr` | `sender` | gated? | +| --- | --- | --- | --- | +| `list_leads` / `get_lead` / `enrich_lead` | ✓ | — | no | +| `mark_disqualified` | ✓ | — | no | +| `draft_email` | ✓ | — | **needsApproval** — the durable pause is the review queue | +| `log_activity` | ✓ | ✓ | no | +| `send_email` | — | ✓ | **needsApproval** + execute() refuses non-`approved` drafts | + +## Phases / checks + +| Phase | Check | +| --- | --- | +| 1 — work the queue | the deterministic operator approves drafts that reference the lead's real signal and **rejects exactly one draft** (first draft for `lead-2`, once) with the feedback note *"too generic — mention their Series B and the async-standup angle"* — the agent must redraft incorporating it | +| 2 — send | the `sender` agent is handed the approved draft ids and sends them; each `send_email` call pauses for approval too | +| 3 — defense in depth | the harness calls the send path directly on `draft-rogue` (status `pending`, never queued) — it must refuse | + +Assertions at the end: the out-of-ICP lead is `disqualified`; every ICP lead +(`lead-1..3`) got an `enrich_lead` call; `approval.requested` events for +`draft_email` equal the decisions made and created drafts equal approved +calls (no draft escaped review); the rejected lead was redrafted mentioning +the feedback; `sent[]` contains only approved drafts; `events.jsonl` plus +per-lead CRM activities record the full enrich → draft → approve → send +chain; the paused approval was persisted under `store/` at pause time. + +## Run + +```bash +cd audit +npx tsx sdr-outreach/index.ts +``` + +`OPENROUTER_API_KEY` must be set (repo-root `.env`). `AUDIT_MODEL` overrides +the model (default `openrouter/openai/gpt-4o-mini`). + +Run artifacts land in `.runs//`: a `leads.json` copy, `store/` (the +durable session + approvals store — `sdr/` and `sender/` subdirs), +`events.jsonl` (the audit log), and `crm-snapshot.json` (final leads, +drafts and sent mail). diff --git a/audit/sdr-outreach/index.ts b/audit/sdr-outreach/index.ts new file mode 100644 index 00000000..8209f81b --- /dev/null +++ b/audit/sdr-outreach/index.ts @@ -0,0 +1,553 @@ +/** + * sdr-outreach — AI-SDR audit harness (live, OpenRouter). + * + * Replicates the production "AI SDR" pattern (Reply.io / AiSDR / Mesh-Pilot + * style — drafts-as-pull-requests): an agent works a CRM lead queue — + * enrich, qualify against the ICP, draft personalized cold outreach — but + * can NEVER send. Every draft stops in a durable human approval queue + * (`needsApproval: true` on draft_email — the pending approval record IS + * the queue entry, persisted by fileStore). A rejection carries reviewer + * feedback back to the model, which redrafts. + * + * Sending is a different principal with a different tool allowlist: a + * second agent whose tools are ONLY send_email + log_activity — the SDR's + * tool list simply omits send_email. send_email is itself approval-gated + * AND its execute() refuses any draft not marked approved (defense in + * depth, exercised in phase 3 against a rogue draft that never went + * through the queue). + * + * cd audit && npx tsx sdr-outreach/index.ts + */ +import '../_shared/env.ts'; +import { appendFileSync, copyFileSync, existsSync, mkdirSync, readFileSync, readdirSync, writeFileSync } from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { z } from 'zod'; +import { createAgent, defineTool, fileStore, type AgentEvent, type ExecutionResult, type PendingApproval } from '@lousho/build-ai-agent'; +import { LIVE_MODEL, hasLiveKey, report } from '../_shared/env.ts'; + +if (!hasLiveKey) { + console.error('OPENROUTER_API_KEY not set (loadAuditEnv found nothing). Bailing.'); + process.exit(2); +} + +const here = path.dirname(fileURLToPath(import.meta.url)); +const stamp = new Date().toISOString().replace(/[:.]/g, '-'); +const runDir = path.join(here, '.runs', stamp); +const storeDir = path.join(runDir, 'store'); +mkdirSync(storeDir, { recursive: true }); +const auditLog = path.join(runDir, 'events.jsonl'); +const log = (entry: Record) => + appendFileSync(auditLog, JSON.stringify({ at: new Date().toISOString(), ...entry }) + '\n'); + +// Self-contained run artifacts: the CRM fixture travels with the run dir. +copyFileSync(path.join(here, 'leads.json'), path.join(runDir, 'leads.json')); + +/* ------------------------------------------------------------------ */ +/* CRM state (in-memory, seeded from leads.json) */ +/* ------------------------------------------------------------------ */ + +interface Activity { + at: string; + type: string; + note: string; +} +interface Lead { + id: string; + name: string; + role: string; + company: string; + industry: string; + size: number; + signal: string; + status: string; + activities: Activity[]; +} +type DraftStatus = 'pending' | 'approved' | 'rejected' | 'sent'; +interface Draft { + id: string; + leadId: string; + subject: string; + body: string; + status: DraftStatus; + createdAt: string; +} + +const leads = new Map(); +for (const l of JSON.parse(readFileSync(path.join(here, 'leads.json'), 'utf8')) as Lead[]) leads.set(l.id, l); + +const drafts = new Map(); +const sent: { draftId: string; leadId: string; to: string; subject: string; at: string }[] = []; +/** Leads enrich_lead ran on — the harness asserts every ICP lead is in here. */ +const enrichments = new Set(); +/** Draft ids actually created by draft_email's execute() (an approved call). */ +const createdDraftIds: string[] = []; +/** Draft ids the reviewer approved (the queue decision recorded on the draft). */ +const approvedDraftIds = new Set(); +let draftSeq = 0; + +// A draft somebody parked in the CRM outside the agent workflow — it never +// went through the approval queue, so it stays 'pending' forever and the +// send path must refuse it (phase 3 checks this directly). +drafts.set('draft-rogue', { + id: 'draft-rogue', + leadId: 'lead-1', + subject: 'quick question', + body: 'Saw the team is hiring — worth a chat?', + status: 'pending', + createdAt: new Date().toISOString(), +}); + +const addActivity = (leadId: string, type: string, note: string) => { + const lead = leads.get(leadId); + if (lead) lead.activities.push({ at: new Date().toISOString(), type, note }); +}; + +/** Simulated enrichment backend, keyed by lead id. */ +const ENRICHMENT: Record = { + 'lead-1': { + techStack: ['Kotlin', 'Kubernetes', 'PostgreSQL'], + recentNews: 'Opened a second engineering hub in Lisbon; hiring page lists 6 backend roles', + headcountGrowth: '+18% engineering headcount in the last 6 months', + mutualConnections: ['J. Park (former colleague, now at Nimbus)'], + }, + 'lead-2': { + techStack: ['TypeScript', 'React', 'AWS'], + recentNews: 'Closed an $18M Series B led by NorthPeak Capital to scale the engineering org', + headcountGrowth: 'plans to grow engineering from 14 to ~30 this year', + mutualConnections: ['S. Adeyemi (angel in the round)'], + }, + 'lead-3': { + techStack: ['Go', 'Temporal', 'GCP'], + recentNews: 'Engineering blog post on cutting meeting load for distributed teams', + headcountGrowth: '+9% engineering headcount YoY', + mutualConnections: [], + }, + 'lead-4': { + techStack: ['Shopify', 'WordPress'], + recentNews: 'Seasonal storefront work; no dedicated engineering hiring', + headcountGrowth: 'flat — 2 developers', + mutualConnections: [], + }, + 'lead-5': { + techStack: ['Python', 'Flask'], + recentNews: 'Won a regional student hackathon', + headcountGrowth: 'n/a — student club', + mutualConnections: [], + }, +}; + +/* ------------------------------------------------------------------ */ +/* Domain tools — the same CRM surface a production AI SDR wraps */ +/* ------------------------------------------------------------------ */ + +const listLeads = defineTool({ + name: 'list_leads', + description: "List leads in the SDR queue. Optional 'status' filter (e.g. 'new'). Returns summaries including the buying signal.", + input: z.object({ status: z.string().optional() }), + async execute({ status }) { + return [...leads.values()] + .filter((l) => !status || l.status === status) + .map((l) => ({ id: l.id, name: l.name, role: l.role, company: l.company, industry: l.industry, size: l.size, signal: l.signal, status: l.status })); + }, +}); + +const getLead = defineTool({ + name: 'get_lead', + description: 'Fetch one lead with its full CRM activity history.', + input: z.object({ id: z.string() }), + async execute({ id }) { + return leads.get(id) ?? { error: `unknown lead ${id}` }; + }, +}); + +const enrichLead = defineTool({ + name: 'enrich_lead', + description: 'Enrich a lead: tech stack, recent company news, headcount growth, mutual connections.', + input: z.object({ id: z.string() }), + async execute({ id }) { + const lead = leads.get(id); + if (!lead) return { error: `unknown lead ${id}` }; + enrichments.add(id); + addActivity(id, 'enriched', 'enrichment pulled (tech stack / news / headcount growth)'); + log({ kind: 'crm', op: 'enrich_lead', leadId: id }); + return { + id, + name: lead.name, + company: lead.company, + ...(ENRICHMENT[id] ?? { techStack: [], recentNews: 'nothing found', headcountGrowth: 'unknown', mutualConnections: [] }), + }; + }, +}); + +const draftEmail = defineTool({ + name: 'draft_email', + description: + 'Create an outbound email draft for a lead. EVERY draft stops in the human approval queue before it can be sent — the call pauses for review. A rejection returns the reviewer note; incorporate it and draft again.', + input: z.object({ + leadId: z.string(), + subject: z.string(), + body: z.string().describe('full email body — must reference this lead’s real signal, never a generic template'), + }), + needsApproval: true, + async execute({ leadId, subject, body }) { + const lead = leads.get(leadId); + if (!lead) return { error: `unknown lead ${leadId}` }; + const draft: Draft = { id: `draft-${++draftSeq}`, leadId, subject, body, status: 'pending', createdAt: new Date().toISOString() }; + drafts.set(draft.id, draft); + createdDraftIds.push(draft.id); + addActivity(leadId, 'draft_created', `draft ${draft.id}: "${subject}"`); + log({ kind: 'crm', op: 'draft_created', draftId: draft.id, leadId, subject }); + return draft; + }, +}); + +/** The actual send — callable directly so phase 3 can prove the guard without a tool call. */ +async function sendDraft(draftId: string): Promise> { + const draft = drafts.get(draftId); + if (!draft) return { sent: false, reason: `unknown draft ${draftId}` }; + if (draft.status !== 'approved') { + log({ kind: 'crm', op: 'send_refused', draftId, status: draft.status }); + return { sent: false, draftId, leadId: draft.leadId, reason: `draft ${draftId} is '${draft.status}' — only approved drafts can be sent` }; + } + const lead = leads.get(draft.leadId); + const local = (s: string) => s.toLowerCase().replace(/[^a-z]+/g, '.').replace(/^\.|\.$/g, ''); + const to = `${local(lead?.name ?? 'unknown')}@${local(lead?.company ?? 'unknown') || 'unknown'}.com`; + draft.status = 'sent'; + sent.push({ draftId, leadId: draft.leadId, to, subject: draft.subject, at: new Date().toISOString() }); + addActivity(draft.leadId, 'sent', `email sent to ${to}: "${draft.subject}"`); + log({ kind: 'crm', op: 'email_sent', draftId, leadId: draft.leadId, to }); + return { sent: true, draftId, leadId: draft.leadId, to, subject: draft.subject }; +} + +const sendEmail = defineTool({ + name: 'send_email', + description: 'Send a draft that has cleared human review. Refuses drafts whose status is not approved.', + input: z.object({ draftId: z.string() }), + needsApproval: true, + async execute({ draftId }) { + return sendDraft(draftId); + }, +}); + +const makeLogActivity = () => + defineTool({ + name: 'log_activity', + description: "Append a note to a lead's CRM activity timeline.", + input: z.object({ + leadId: z.string(), + type: z.string().describe("activity type, e.g. 'researched', 'note'"), + note: z.string(), + }), + async execute({ leadId, type, note }) { + if (!leads.has(leadId)) return { error: `unknown lead ${leadId}` }; + addActivity(leadId, type, note); + log({ kind: 'crm', op: 'log_activity', leadId, type }); + return { ok: true }; + }, + }); + +const markDisqualified = defineTool({ + name: 'mark_disqualified', + description: 'Mark a lead disqualified with a one-line reason. Use for leads outside the ICP.', + input: z.object({ leadId: z.string(), reason: z.string() }), + async execute({ leadId, reason }) { + const lead = leads.get(leadId); + if (!lead) return { error: `unknown lead ${leadId}` }; + lead.status = 'disqualified'; + addActivity(leadId, 'disqualified', reason); + log({ kind: 'crm', op: 'mark_disqualified', leadId, reason }); + return { ok: true, id: leadId, status: lead.status }; + }, +}); + +/* ------------------------------------------------------------------ */ +/* Agents — the allowlist boundary IS the tool list */ +/* ------------------------------------------------------------------ */ + +const SDR_INSTRUCTIONS = `You are an AI SDR for FlowBoard, an async-standup tool for engineering teams: it replaces the synchronous daily standup with async written check-ins, built for remote and distributed engineering orgs. + +ICP: engineering leaders (Engineering Manager / Director / VP Eng) at tech companies of roughly 50-500 employees showing a hiring or remote-team signal. +Out of ICP: students, tiny non-technical teams, companies far outside that size band. + +Workflow: +1. list_leads to see the queue (status 'new'). The summary already carries role, size and signal — call get_lead only if you need the activity history. +2. For each lead, call enrich_lead to pull tech stack, news, headcount growth, mutual connections. +3. ICP fit -> draft_email with a personalized opener that references THIS lead's real signal or enrichment (their company, the raise, the remote team, the standup pain). Generic templates get rejected by the reviewer — a rejection returns the note; incorporate it and draft_email again. Then log_activity type 'researched' with a one-line rationale. +4. Qualify BEFORE drafting — never draft for a lead you are about to disqualify. Not ICP -> mark_disqualified with a one-line reason. +5. You have NO send capability — drafts go to the human review queue and a separate operator/agent sends what is approved. Never ask for send_email. +Work the whole queue, then reply with one line per lead: qualified/drafted or disqualified and why.`; + +const sdr = createAgent({ + name: 'sdr', + model: LIVE_MODEL, + store: fileStore(path.join(storeDir, 'sdr')), + instructions: SDR_INSTRUCTIONS, + tools: [listLeads, getLead, enrichLead, draftEmail, makeLogActivity(), markDisqualified], + maxSteps: 25, +}); +const sdrSession = sdr.session({ id: 'sdr-queue-1' }); + +const sender = createAgent({ + name: 'sender', + model: LIVE_MODEL, + store: fileStore(path.join(storeDir, 'sender')), + instructions: + 'You are the outbound sender for the SDR team. The operator gives you a list of approved draft ids. ' + + 'For each one: call send_email with the draftId. If a send is refused, report it and move on — do not retry and do not argue. ' + + 'When done, reply with one line per draft: sent or refused.', + tools: [sendEmail, makeLogActivity()], + maxSteps: 15, +}); +const senderSession = sender.session({ id: 'sender-1' }); + +/* ------------------------------------------------------------------ */ +/* Event log */ +/* ------------------------------------------------------------------ */ + +const counts: Record = {}; +const approvalsRequestedByTool: Record = {}; +const short = (v: unknown, n = 160) => { + const s = typeof v === 'string' ? v : JSON.stringify(v); + return s && s.length > n ? s.slice(0, n) + '…' : s; +}; +function show(e: AgentEvent): void { + counts[e.type] = (counts[e.type] ?? 0) + 1; + switch (e.type) { + case 'step.start': console.log(` -- step ${e.step}`); break; + case 'tool.start': console.log(` tool.start ${e.toolName} ${short(e.args, 110)}`); break; + case 'tool.done': console.log(` tool.done ${e.toolName} (${e.durationMs}ms)`); break; + case 'tool.error': console.log(` tool.error ${e.toolName}: ${short(e.error.message, 140)}`); break; + case 'approval.requested': + approvalsRequestedByTool[e.toolName] = (approvalsRequestedByTool[e.toolName] ?? 0) + 1; + console.log(` approval.requested ${e.toolName}`); + break; + case 'text.done': console.log(` text: ${short(e.text, 300)}`); break; + case 'error': console.log(` ERROR ${e.error.name}: ${short(e.error.message, 160)}`); break; + case 'provider.retry': console.log(` provider.retry ${e.attempt}`); break; + case 'run.done': console.log(` == run.done ${e.finishReason}`); break; + } +} +sdrSession.on(show); +senderSession.on(show); + +/* ------------------------------------------------------------------ */ +/* Deterministic operator policy (the human review queue) */ +/* ------------------------------------------------------------------ */ + +// lead-2's signal is the $18M Series B — the one draft this harness always +// rejects once, with actionable feedback, to exercise the redraft loop. +const REJECT_LEAD_ID = 'lead-2'; +const REJECT_NOTE = 'too generic — mention their Series B and the async-standup angle'; +const rejectedOnce = new Set(); +const rejectionsPerLead = new Map(); +const draftCallsPerLead = new Map(); +let draftDecisions = 0; +let draftApprovedCalls = 0; +let draftRejectedCalls = 0; + +const SIGNAL_KEYWORDS: Record = { + 'lead-1': ['nimbus', 'hiring', 'remote', 'time zone', 'backend'], + 'lead-2': ['cartloop', 'series b', 'doubl', 'engineering org'], + 'lead-3': ['ledgerly', 'remote', 'standup', 'maker time', 'meeting'], + 'lead-4': ['petal', 'stem'], + 'lead-5': ['university', 'student', 'hack'], +}; + +function personalized(lead: Lead, text: string): boolean { + const hay = text.toLowerCase(); + const keys = [lead.company.split(/[\s&]+/)[0]!, ...(SIGNAL_KEYWORDS[lead.id] ?? [])]; + return keys.some((k) => k && hay.includes(k.toLowerCase())); +} + +function decide(pending: PendingApproval | undefined): { approved: boolean; note?: string } { + if (!pending) return { approved: false, note: 'no pending approval found' }; + + if (pending.toolName === 'draft_email') { + draftDecisions++; + const leadId = String(pending.args.leadId ?? ''); + draftCallsPerLead.set(leadId, (draftCallsPerLead.get(leadId) ?? 0) + 1); + const lead = leads.get(leadId); + if (!lead) { + draftRejectedCalls++; + return { approved: false, note: `unknown lead ${leadId}` }; + } + // The one scripted rejection: first draft for REJECT_LEAD_ID, once only. + if (leadId === REJECT_LEAD_ID && !rejectedOnce.has(leadId)) { + rejectedOnce.add(leadId); + draftRejectedCalls++; + return { approved: false, note: REJECT_NOTE }; + } + const text = `${pending.args.subject ?? ''}\n${pending.args.body ?? ''}`; + if (personalized(lead, text)) { + draftApprovedCalls++; + return { approved: true }; + } + // Non-personalized drafts get rejected — but a reviewer can't stall the + // queue forever, so each lead gets at most one quality rejection. + const n = rejectionsPerLead.get(leadId) ?? 0; + if (n < 1) { + rejectionsPerLead.set(leadId, n + 1); + draftRejectedCalls++; + return { approved: false, note: `reviewer: reads like a template — reference ${lead.company}'s actual signal (${lead.signal})` }; + } + console.log(` !! operator override: approving non-personalized draft for ${leadId} after rejection budget`); + draftApprovedCalls++; + return { approved: true }; + } + + if (pending.toolName === 'send_email') { + const draftId = String(pending.args.draftId ?? ''); + const draft = drafts.get(draftId); + if (draft?.status === 'approved') return { approved: true }; + return { approved: false, note: `operator policy: draft ${draftId} is not approved — send refused` }; + } + + return { approved: false, note: `operator policy: unexpected tool ${pending.toolName}` }; +} + +/** The reviewer's queue decision, recorded on the draft and in the CRM. */ +function flushApprovedDrafts(): void { + for (const d of drafts.values()) { + if (d.status === 'pending' && createdDraftIds.includes(d.id)) { + d.status = 'approved'; + approvedDraftIds.add(d.id); + addActivity(d.leadId, 'draft_approved', `reviewer approved ${d.id}`); + log({ kind: 'crm', op: 'draft_approved', draftId: d.id, leadId: d.leadId }); + } + } +} + +const approvalFileExists = (id: string) => + ['sdr', 'sender'].some((sub) => existsSync(path.join(storeDir, sub, 'approvals', `${id}.json`))); + +let durablePauseSeen = false; + +async function drive( + agent: ReturnType, + send: (input: string) => Promise, + input: string +): Promise { + let result = await send(input); + while (result.finishReason === 'awaiting-approval' && result.approvalId) { + if (approvalFileExists(result.approvalId)) durablePauseSeen = true; + const pending = await agent.approvals.get(result.approvalId); + const d = decide(pending); + console.log(` >> operator ${d.approved ? 'APPROVED' : 'REJECTED'} ${pending?.toolName} ${short(pending?.args, 100)}${d.note ? ` (${d.note})` : ''}`); + log({ kind: 'approval-decision', id: result.approvalId, tool: pending?.toolName, args: pending?.args, ...d }); + if (!d.approved && pending?.toolName === 'draft_email') { + addActivity(String(pending.args.leadId ?? ''), 'draft_rejected', `reviewer: ${d.note ?? 'rejected'}`); + } + result = await agent.approvals.resolve({ id: result.approvalId, approved: d.approved, ...(d.note && { note: d.note }) }); + flushApprovedDrafts(); + } + return result; +} + +/* ------------------------------------------------------------------ */ +/* Run */ +/* ------------------------------------------------------------------ */ + +console.log(`model: ${LIVE_MODEL}\nrun dir: ${runDir}\nstore: ${storeDir}`); + +console.log('\n=== PHASE 1: the SDR works the lead queue ==='); +const t1 = await drive( + sdr, + (m) => sdrSession.send(m), + 'Work the lead queue: research each lead, draft personalized outreach for ICP-fit leads, disqualify the rest.' +); +log({ kind: 'phase', phase: 'sdr', finishReason: t1.finishReason }); +console.log(`phase 1 finish: ${t1.finishReason}`); + +console.log('\n=== PHASE 2: a different principal sends the approved drafts ==='); +const approved = [...drafts.values()].filter((d) => d.status === 'approved'); +log({ kind: 'phase', phase: 'send', approvedDrafts: approved.map((d) => d.id) }); +const sendList = approved + .map((d) => `${d.id} (lead ${d.leadId}, ${leads.get(d.leadId)?.name} @ ${leads.get(d.leadId)?.company})`) + .join('; '); +const t2 = await drive( + sender, + (m) => senderSession.send(m), + approved.length + ? `These drafts cleared human review: ${sendList}. Send each one with send_email, then report the outcome per draft.` + : 'No drafts cleared review. Reply with "nothing to send".' +); +log({ kind: 'phase', phase: 'send-done', finishReason: t2.finishReason }); +console.log(`phase 2 finish: ${t2.finishReason}`); + +console.log('\n=== PHASE 3: defense in depth — the send path refuses unapproved drafts ==='); +// draft-rogue never entered the queue; even a call that bypassed the approval +// layer entirely must fail inside send_email's execute(). +const rogueAttempt = await sendDraft('draft-rogue'); +report( + 'send path refuses an unapproved draft', + rogueAttempt.sent === false && !sent.some((s) => s.draftId === 'draft-rogue'), + `result=${short(rogueAttempt, 160)}` +); + +/* ------------------------------------------------------------------ */ +/* Checks */ +/* ------------------------------------------------------------------ */ + +console.log('\n=== VERDICT ==='); + +const lead5 = leads.get('lead-5'); +report('out-of-ICP lead disqualified', lead5?.status === 'disqualified', `lead-5 status=${lead5?.status} activities=${lead5?.activities.length}`); + +const icpIds = ['lead-1', 'lead-2', 'lead-3']; +report( + 'every ICP lead was enriched', + icpIds.every((id) => enrichments.has(id)), + `enriched=${[...enrichments].sort().join(',')}` +); + +const requestedDraft = approvalsRequestedByTool['draft_email'] ?? 0; +report( + 'every draft paused in the human approval queue', + requestedDraft > 0 && requestedDraft === draftDecisions && createdDraftIds.length === draftApprovedCalls, + `approval.requested(draft_email)=${requestedDraft} decisions=${draftDecisions} approved=${draftApprovedCalls} rejected=${draftRejectedCalls} created=${createdDraftIds.length}` +); + +const lead2Drafts = [...drafts.values()].filter((d) => d.leadId === REJECT_LEAD_ID); +const incorporatesNote = lead2Drafts.some((d) => /series b|async/i.test(`${d.subject} ${d.body}`)); +report( + 'rejected draft was redrafted with the feedback incorporated', + rejectedOnce.has(REJECT_LEAD_ID) && (draftCallsPerLead.get(REJECT_LEAD_ID) ?? 0) >= 2 && incorporatesNote, + `lead-2 draft_email calls=${draftCallsPerLead.get(REJECT_LEAD_ID) ?? 0} drafts=${lead2Drafts.length} mentionsNote=${incorporatesNote}` +); + +const sentClean = sent.length > 0 && sent.every((s) => approvedDraftIds.has(s.draftId)); +const allApprovedSent = [...approvedDraftIds].every((id) => sent.some((s) => s.draftId === id)); +report( + 'only approved drafts were sent', + sentClean && allApprovedSent, + `sent=${sent.length} approved=${approvedDraftIds.size} rogueSent=${sent.some((s) => s.draftId === 'draft-rogue')}` +); + +const sentLeadIds = new Set(sent.map((s) => s.leadId)); +const chainOk = + sentLeadIds.size > 0 && + [...sentLeadIds].every((id) => { + const types = new Set(leads.get(id)?.activities.map((a) => a.type) ?? []); + return ['enriched', 'draft_created', 'draft_approved', 'sent'].every((t) => types.has(t)); + }); +const auditLines = existsSync(auditLog) ? readFileSync(auditLog, 'utf8').trim().split('\n').filter(Boolean).length : 0; +report( + 'audit trail: events.jsonl + CRM activities record the full chain', + chainOk && auditLines > 5, + `auditLines=${auditLines} sentLeads=${[...sentLeadIds].join(',')}` +); + +const storeArtifacts = (sub: string) => (existsSync(path.join(storeDir, sub)) ? readdirSync(path.join(storeDir, sub), { recursive: true }).length : 0); +report( + 'durable: paused approvals persisted under store/', + durablePauseSeen && storeArtifacts('sdr') > 0, + `durablePauseSeen=${durablePauseSeen} sdrStoreEntries=${storeArtifacts('sdr')} senderStoreEntries=${storeArtifacts('sender')}` +); + +writeFileSync( + path.join(runDir, 'crm-snapshot.json'), + JSON.stringify({ leads: [...leads.values()], drafts: [...drafts.values()], sent }, null, 2) +); +console.log('event counts:', JSON.stringify(counts)); +console.log(`run dir kept at ${runDir}`); diff --git a/audit/sdr-outreach/leads.json b/audit/sdr-outreach/leads.json new file mode 100644 index 00000000..b90a5621 --- /dev/null +++ b/audit/sdr-outreach/leads.json @@ -0,0 +1,61 @@ +[ + { + "id": "lead-1", + "name": "Priya Nair", + "role": "Engineering Manager", + "company": "Nimbus Health", + "industry": "healthtech SaaS", + "size": 220, + "signal": "Hiring 6 backend engineers; team is fully remote across 4 time zones", + "status": "new", + "activities": [ + { "at": "2025-01-06T09:12:00Z", "type": "imported", "note": "Inbound from SaaStr conference list" } + ] + }, + { + "id": "lead-2", + "name": "Marcus Feld", + "role": "VP of Engineering", + "company": "Cartloop", + "industry": "e-commerce SaaS", + "size": 140, + "signal": "Just raised an $18M Series B; doubling the engineering org this year", + "status": "new", + "activities": [ + { "at": "2025-01-07T14:40:00Z", "type": "imported", "note": "Sourced from funding-announcement feed" } + ] + }, + { + "id": "lead-3", + "name": "Sofia Tran", + "role": "Director of Platform Engineering", + "company": "Ledgerly", + "industry": "fintech", + "size": 380, + "signal": "Remote-first engineering org; posted publicly about daily standups eating maker time", + "status": "new", + "activities": [] + }, + { + "id": "lead-4", + "name": "Dan Okafor", + "role": "CTO", + "company": "Petal & Stem", + "industry": "florist retail", + "size": 12, + "signal": "Two-person dev team; evaluating cheaper project management tools", + "status": "new", + "activities": [] + }, + { + "id": "lead-5", + "name": "Alex Rivera", + "role": "Student", + "company": "State University Hack Club", + "industry": "education", + "size": 3, + "signal": "Building a class project; wants a free tool for a 3-person student team", + "status": "new", + "activities": [] + } +]