Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
9 changes: 9 additions & 0 deletions audit/eve/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Eve audit run debris — repro scripts and findings are committed, runtime output is not.
*.log
*.db
*.sqlite*
.tmp*/
out-*/
dist/
node_modules/
.runs/
92 changes: 92 additions & 0 deletions audit/eve/00-live-confirmed-resume.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# EVE-0 — `agent.resume(id)` silently no-ops on a paused channel session

**Severity: high** | **Status: FIXED upstream — re-verified on main @ f4c8eb10**

The fix rounds added a session-level paused marker (`<id>.paused.json`,
AgentSession.ts `pausedTurnKey`/`pausedSessionTurn`), so a pending
approval is discoverable regardless of transcript bookkeeping. All three
repros now correctly throw `SessionAwaitingApprovalError`; the stale
marker is lazily deleted on the next turn. (Original report below.)

Found during live re-verification: `incident-responder` run against
`openrouter/openai/gpt-4o-mini` — the durable-store check
"session checkpoint marked awaiting-approval (resume refuses)" FAILED:
`commander.resume('inc-7001')` resolved `null` while
`checkpoints/inc-7001.turn-0.json` sat on disk with
`status: "awaiting-approval"`.

## Root cause

`mountChannels(agent, [channel])` without `options.store` gives channel
turns a **MemorySessionStore transcript** (channelCore.ts:101), while
`withDefaultStores` backfills `checkpointStore`/`approvalStore` from the
agent's durable `fileStore`.

The approval-continuation path (`approvals.resolve` → resume of the
checkpointed turn) commits the finished transcript through a session bound
to the **durable** sessions store. Net effect after the first completed
channel turn:

- `sessions/<id>.json` on disk = the finished transcript (length L > 0),
frozen — the live channel keeps writing to memory.
- Every subsequent channel turn sees transcript length 0 → checkpoints
`<id>.turn-0` (overwriting the previous turn's checkpoint each time).
- `agent.resume(id)` → `session({id}).resume()` → `pendingCheckpoint()`
derives `turn-<L>` from the **durable** transcript → no such file →
returns `null`. The real awaiting-approval checkpoint at `turn-0` is
invisible to it.

`loadRunState`'s `SessionAwaitingApprovalError` guard (LOU-U8) is never
reached. No error, no signal — the caller just gets `null` and the paused
turn is silently abandoned.

## Wider blast radius (same root cause)

The pause is discoverable only via `<id>.turn-<transcript.length>` under
the *caller's* transcript view — there is no session-level "paused"
marker. Verified siblings:

- `repro/paused-send-overlap.mjs`: `agent.send(msg, {sessionId:'chat'})`
while `chat.turn-0` awaits approval → **runs a full second turn**
(`finishReason: stop`), leaving `chat.json` + `chat.turn-0.json`
coexisting in the checkpoint dir. The paused approval is not a lock.
- `repro/paused-session-send.mjs`: `session({id:'chat'}).send(...)` on the
durable view while the channel turn is paused → runs a new turn and
commits to the durable transcript, bypassing the pending approval
entirely.

So "paused for approval" means only: *the same caller, using the same
transcript store, with the same transcript length* will see it. Everyone
else can run new work under the same session id mid-approval.

## Secondary damage

- Transcript divergence is permanent: durable `sessions/<id>.json` never
sees later channel turns; the channel's memory transcript is lost on
restart. "Durable sessions" is false for channel traffic by default.
- `turn-0` checkpoint gets clobbered by each new turn, destroying the
checkpoint-history/fork story for the session.

## Repro

`audit/eve/repro/resume-divergence.mjs` — mock model, no network:
1. Completed turn on durable session `chat` → `sessions/chat.json` (2 msgs).
2. Channel-shaped session (`agent.session({id:'chat', store: new MemorySessionStore()})`)
pauses on a `needsApproval` tool → `chat.turn-0` awaiting-approval on disk.
3. `agent.resume('chat')` → **resolved null** (expected
`SessionAwaitingApprovalError`).

## Fix direction

Either:

- `resume()`/`pendingCheckpoint()` should consult the checkpoint store's
pending entries for the session prefix (`<id>.turn-*`) rather than only
`<id>.turn-<transcript.length>` — an awaiting-approval checkpoint should
ALWAYS produce `SessionAwaitingApprovalError` regardless of transcript
bookkeeping; **and/or**
- channel sessions should default to the agent's durable transcript store
so the two views can't diverge (the in-memory default exists so channel
demos work without a store — but silently diverging from the checkpoint
store is the bug); at minimum `mountChannels` should warn when the agent
is durable and no channel store is given.
72 changes: 72 additions & 0 deletions audit/eve/BRIEF.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# Eve adversarial audit — brief (round 2, 2026-10-09)

You are a senior engineer on the **Vercel Eve** team. Your job is to build the
strongest *honest* case that `@lousho/build-ai-agent` is **not** a good SDK to
bet a production agent on. Hostile but fair: every claim must be verified by
running code or citing `src/...:line`. Unverified opinions are worthless —
the owner will use this to fix the SDK, so false positives waste their time.

## Four lenses — every feature in your area gets all four

1. **AI engineer** — does it behave correctly with real models? Prompt/tool
schema quality, model-facing descriptions, token waste, failure handling
(malformed tool calls, refusals, truncation, provider 4xx/5xx, rate limits),
streaming correctness, determinism, evals/observability of model behavior.
2. **Product manager** — does it solve the job a buyer has? Missing
table-stakes vs Vercel AI SDK / Eve, OpenAI Agents SDK, Mastra, LangGraph,
Claude Agent SDK. Is the feature discoverable, documented, worth its
complexity? Would a team pick this over the alternatives?
3. **Software engineer** — correctness, concurrency, durability, security,
types (`tsc --noEmit` under zod 4 with skipLibCheck: true is the baseline),
API consistency, error codes/messages, Windows behavior, test gaps.
4. **Designer (DX / API design / UX)** — naming, defaults, progressive
disclosure, error message quality, docs examples that actually run,
consistency across surfaces, for UI pieces: actual visual/interaction UX
and accessibility.

## Environment

- Workspace `E:\agent-sdk\audit` with the **fresh packed tarball from main**
(`1.0.0-alpha.19`, installed in `audit/node_modules`). Import only from
`@lousho/build-ai-agent` and its subpaths — never `../../src`.
- Live model: `OPENROUTER_API_KEY` is in `E:\agent-sdk\.env`. Import
`../_shared/env.ts` (loads it; exports `LIVE_MODEL` =
`openrouter/openai/gpt-4o-mini`, `hasLiveKey`, `report()`). Keep live calls
cheap (gpt-4o-mini, small prompts, < ~150 calls total for your area).
Prefer deterministic repros (mock/scripted providers from
`@lousho/build-ai-agent/testing` if exported) where a live model isn't needed.
- Run scripts with `npx tsx <file>` from `E:\agent-sdk\audit`.
- SDK source `E:\agent-sdk\src`, docs `E:\agent-sdk\docs`, examples
`E:\agent-sdk\examples`, apps `E:\agent-sdk\apps`.
- **Do not modify anything outside `audit/eve/`.** Never print secrets.
Never mutate real infrastructure, send real messages, or spend money beyond
the cheap OpenRouter calls.

## Already reported — do NOT re-report (read them to avoid duplicates)

- `audit/eve/00-live-confirmed-resume.md` (EVE-0 resume no-op on channel sessions)
- `audit/eve/FINDINGS-channels-server.md`
- `audit/eve/FINDINGS-cli-onboarding.md`
- `audit/eve/FINDINGS-memory-context.md`
- `audit/REPORT.md` (prior audit; its "fixed" items are claimed fixed — you
MAY report a regression if you prove a claimed fix doesn't hold).

## Output

Write `audit/eve/FINDINGS-<area>.md`. Repros go in `audit/eve/<area>/`.
Format:

```
# Eve audit: <area>
## Summary (3-6 bullets: the pitch for why this area is weak)
## Findings
### <AREA>-F<n>: <title>
**Severity** critical/high/medium/low · **Lens** AI eng / PM / SWE / Design · **Type** bug/DX/docs/enhancement/perf/security
**Evidence** — command + observed output (trimmed) or repro path
**Root cause** — `src/...:line`
**Suggested fix** — concrete, 1-point-ticket sized where possible
## Verified non-issues (checked, holds up)
```

Aim for depth over breadth: 8-20 real findings. Rank by severity. Final reply
to the orchestrator: a terse table of finding IDs, titles, severities.
Loading
Loading