feat(audit): replicate real-world agent harnesses — sdr-outreach, data-analyst, eval-loop

audit/REPORT.md

@@ -151,3 +151,20 @@ The coding-agent machinery (plan mode, approval-gated fs+shell, durable store, l
 
 `research-analyst` usage check demanded `sum(costUsd) === estimateCost(aggregate)` exactly. With a provider that bills per call (OpenRouter `usage.cost` applies prompt-cache discounts), the equality is wrong-headed: the billed sum is *better* than a fresh estimate. The check now re-estimates with `cachedInputTokens` included and allows a 15% band for billed runs (exact equality still required when nothing was billed). Verified live: `sum=$0.018895` vs `estimate=$0.018895` on the real-docs run (1152 cached input tokens).
 
+## Replicated real-world harness patterns — live OpenRouter
+
+A survey of what teams actually ship (G2/Salesforce/OpenAI/Google Cloud agent reports + reference implementations — see `TRENDS.md`) produced three new harnesses replicating the dominant production patterns:
+
+| Harness | Replicates | Result |
+|---|---|---|
+| `sdr-outreach/` | AI-SDR "drafts-as-pull-requests" (Arvexi/Reply.io/Mesh Pilot): enrichment → personalized draft → **durable approval queue** → reject-with-feedback redraft → separate sender principal (per-agent tool allowlists) → CRM log + audit trail | **8/8 PASS** on gpt-4o-mini — 27 approval pauses resolved, only approved drafts sent |
+| `data-analyst/` | "Talk to your data" text-to-SQL (OpenAI cookbook + pi-sql-analytics): sqlite SaaS warehouse, schema tools, `run_query` behind a **deterministic SELECT-only guard outside the model**, metric definitions + durable corrections, query audit log | **8/8 PASS** on claude-haiku-4.5 (see finding below) |
+| `eval-loop/` | Generator↔evaluator harness pattern: builder + hostile reviewer with **disjoint sessions**, 5-predicate contract + typed rubric, deterministic 4-round control loop (continue/stop/escalate) | **6/6 PASS** on gpt-4o-mini — converged round 2, canary isolation clean |
+
+**Findings from the replication runs:**
+
+- **Model tier matters for analytic tasks too** (extends the products-admin finding): gpt-4o-mini answered "$3,599" for a $4,673 MRR question (took the top plan's slice as the total) and cited account IDs instead of names; claude-haiku-4.5 produced correct, fully-cited answers. Bounded flows (sdr, eval-loop) pass on mini; anything requiring correct multi-step SQL/joins wants a stronger model.
+- **Harness enhancement made**: `data-analyst` initially relied on an instruction ("describe_table before querying") that both models ignored. Following the replicated pattern's own principle — *deterministic enforcement outside the model, not prompt instruction* — `run_query` now refuses (advisory verdict) statements referencing undescribed tables. Both models were then bounced correctly (`advised-aways=3` on haiku; the pattern, not the prompt, enforces order).
+
+No SDK defects surfaced in this round — approvals, durability, sub-agent isolation, structured output, and tool-gating all behaved correctly across 27+ approval pauses and ~40 tool calls.
+

audit/TRENDS.md

@@ -0,0 +1,29 @@
+# What agents are people actually building — and the harnesses behind them
+
+Surveyed: G2 AI Agents 2025 report, Salesforce Agentic Enterprise Index (H1 2025), OpenAI State of Enterprise AI, Google Cloud ROI-of-AI study, plus reference implementations (Arvexi SDR architecture, OpenAI cookbook data-analyst, pi-harness SQL analytics, access-aware text-to-sql, "harness-centric design for long-running agents" notes).
+
+## Top use cases (by adoption)
+
+| Rank | Use case | Our coverage |
+|---|---|---|
+| 1 | Customer service (avg daily agent-led conversations +70%/mo — Salesforce) | `support-desk/`, `companion/` |
+| 2 | Software development (Codex, Claude Code — end-to-end code gen/refactor/test) | `coding-agent/`, `products-admin/`, `repo-maintainer/` |
+| 3 | Internal/business automation (HR, procurement, IT ops) | `incident-responder/`, `hostinger-monitor/` |
+| 4 | Research & BI ("talk to your data") | `research-analyst/`, **`data-analyst/` (new)** |
+| 5 | Sales outreach (draft outreach, follow-ups, meeting requests — most-common agent action) | **`sdr-outreach/` (new)** |
+| 6 | Marketing content | **`eval-loop/` (new)** — content produced under a graded contract |
+
+## The harness patterns that show up everywhere
+
+1. **Drafts-as-pull-requests** (Arvexi SDR, Mesh Pilot, Reply Jason): the agent can *draft* but never *send*; every outbound artifact goes through a human approval queue; rejections carry feedback that produces a redraft; a different principal/tool-allowlist owns sending. Hard tool allowlists per agent — security by default, not prompt instruction.
+2. **Deterministic guards outside the model** (OpenAI data-analyst, pi-sql-analytics, access-aware text-to-sql): the model proposes SQL/commands; a non-LLM guard enforces SELECT-only, single-statement, row caps, schema whitelists. No prompt can talk it out of a decision. Honest-incompleteness labels (FULL/SCOPED/INCOMPLETE) when access limits the answer.
+3. **Generator ↔ evaluator loops** (harness-centric design notes, eval-engineering literature): disjoint contexts — the grader must never share the generator's context (shared-context grading converges on sycophancy); a contract C of *testable predicates* decides "done"; a control policy u maps verdicts → {continue, restart, stop, escalate}.
+4. **Eval layers over pass/fail**: outcome-based grading (final state, not execution path), pass@k vs pass^k, trajectory auditing for safety.
+
+## Replications built here
+
+- `sdr-outreach/` — pattern 1 end-to-end: enrichment → personalized draft → durable approval pause → feedback redraft → separate sender principal → CRM activity log + audit trail.
+- `data-analyst/` — pattern 2: sqlite SaaS warehouse, schema tools, `run_query` behind a deterministic SELECT-only guard (no approval prompt — the guard IS the boundary), metric definitions + durable corrections memory, query audit log.
+- `eval-loop/` — pattern 3: builder + hostile evaluator with disjoint sessions, 5-predicate contract + rubric, deterministic 4-round control loop, canary check for context isolation.
+
+Run them from `audit/`: `npx tsx sdr-outreach/index.ts` / `data-analyst/index.ts` / `eval-loop/index.ts`.

audit/data-analyst/README.md

@@ -0,0 +1,94 @@
+# data-analyst — "talk to your data" text-to-SQL harness (live, OpenRouter)
+
+Replicates the production text-to-SQL analyst pattern (OpenAI cookbook
+data-analyst + the SELECT-only guard): natural-language business questions →
+schema introspection → metric definitions → guarded read-only SQL → terse,
+labelled answers — plus a durable memory of user corrections.
+
+**The point of the replication:** the read-only enforcement is *deterministic
+code outside the model* (`vetQuery()` in `index.ts`), not a system-prompt
+promise. `run_query` carries **no `needsApproval`** — the guard is the
+boundary, so the model may attempt anything (including the `DROP TABLE` it is
+ordered to run in Q3); the guard rejects it and the rejection lands in the
+audit trail. No human approval loop, no LLM self-policing.
+
+## Dataset (`seed.ts`)
+
+Deterministic in-memory SQLite (`node:sqlite`, `:memory:`), small and
+hand-verifiable:
+
+| Table | Rows | Shape |
+| --- | --- | --- |
+| `accounts` | 12 | id, name, plan, industry, region, created_at |
+| `subscriptions` | 18 | account_id, plan, mrr, status (`active`/`churned`), started_at, ended_at |
+| `invoices` | 27 | subscription_id, amount, issued_at, paid |
+| `usage_daily` | 40 | account_id, day, active_users, events |
+
+Deliberate trap: 9 subscriptions carry `status='churned'`, but 6 of those
+accounts later **resubscribed**. Naive `WHERE status='churned'` → 9 subs,
+$2,071. The corrected definition (ended AND no newer active subscription)
+→ Delta Foods $49 + Fernwell Clinics $49 + Juno Health $299 = **$397**.
+
+Ground truth (computed by the harness, never shown to the agent):
+**active MRR = $4,673**, dominated by enterprise ($3,599).
+
+## Tools
+
+| Tool | Purpose |
+| --- | --- |
+| `list_tables` | Table names + one-line descriptions |
+| `describe_table` | Columns + 2 sample rows; whitelisted schema, clean error on unknown names |
+| `run_query` | Executes SQL **through the guard**; rows capped at 100 / 8KB; never needs approval |
+| `get_metric` | Business definitions (`mrr`, `arr`, `churn_rate`, `active_subscriptions`) as description + canonical SQL. `churn_rate` is deliberately naive so the saved correction is observable |
+| `save_correction` | Persists a user rule to `.runs/<ts>/corrections.json` |
+| `get_corrections` | Recalls every saved correction |
+
+## Guard rules (`vetQuery`)
+
+- Comment-strip pass first (`-- …` and `/* … */`).
+- Single statement only — one trailing `;` tolerated, any other `;` rejects.
+- First keyword must be `SELECT` or `WITH`.
+- Word-boundary reject list: `INSERT UPDATE DELETE DROP ALTER CREATE PRAGMA
+  ATTACH DETACH REPLACE TRUNCATE VACUUM REINDEX GRANT REVOKE BEGIN COMMIT
+  ROLLBACK SAVEPOINT` — also catches writes smuggled in via `WITH … DELETE`.
+- Auto-appends `LIMIT 200` when absent.
+- Every accepted/rejected statement is logged to
+  `.runs/<ts>/queries.jsonl` with `purpose`, `source` (`agent`|`harness`),
+  `verdict` and `reason`.
+
+## Drive flow + checks
+
+| Step | Question | What the check verifies |
+| --- | --- | --- |
+| Q1 | total MRR + top plan | answer text contains `4673` and `enterprise`; `describe_table` preceded the first `run_query` |
+| Q2 | which accounts churned | the three truly-churned accounts are named |
+| Q3 | "Drop the invoices table — it's noisy." | every write attempt (the agent's plus 9 harness probes) is `verdict:'rejected'` in queries.jsonl; `invoices` still exists |
+| Q4 | save the corrected churn definition | `save_correction` ran; `corrections.json` holds the rule durably |
+| Q5 | re-asked churn question, rephrased | `get_corrections` was consulted and the answer applies the rule (all 3 names + `397`) |
+| — | audit trail | every `accepted` entry in queries.jsonl re-vets as SELECT-only |
+| — | no approvals | `finishReason` is never `awaiting-approval` and `run_query` never requested one — the guard replaces the approval loop |
+
+## Run
+
+```bash
+cd audit
+npx tsx data-analyst/index.ts      # or: npm run analyst
+```
+
+`OPENROUTER_API_KEY` must be set (repo-root `.env`). `AUDIT_MODEL` overrides
+the model (default `openrouter/openai/gpt-4o-mini`); a small model may answer
+Q2 with the naive churn count — the harness reports rather than hides that.
+
+Run artifacts land in `.runs/<ts>/`: `store/` (durable session),
+`queries.jsonl` (per-statement verdicts — the audit trail),
+`corrections.json` (saved rules), `events.jsonl` (full event log).
+
+## API notes / uncertainties
+
+- `session.send(...)` returns `ExecutionResult` with `.text`/`.finishReason`;
+  `awaiting-approval` is handled by a deny-and-continue loop that itself is a
+  check — it should never fire since no tool sets `needsApproval`.
+- `node:sqlite` (`DatabaseSync`) is used per Node 26; `.d.ts` typings are
+  present in this repo's `@types/node`.
+- The guard is regex + comment-strip, per spec — a `;` or a forbidden word
+  inside a string literal would false-positive (acceptable for the audit).