[TheGreatAxios]

First step of CL-10452 (wire @corbits/evals + trajectory scorers).

What

• Add @corbits/evals@0.1.1 as a devDependency.
  • License note: LGPL-2.1-only (see https://github.com/corbitsdev/corbits-evals#readme); deps @corbits/system-one@^0.3.0 (peer @intx/workflow optional). Dev-time eval harness only, never bundled into the product.
• Land the first eval built on it: dispatch-routing.
  • Exercises the real dispatch director system prompt (agents/dispatch/src/prompt.ts, exported by @corbits/code-agent-dispatch) against 9 brand-new routing requests (evals/prompts/dispatch-routing/cases.jsonl).
  • Runs through the package evaluate() with checks (mentions routes, avoids DIY, under 120s) and allowed-miss gating (routes: miss 6, noDiy: miss 2, fast: 0), plus compare/save drop gating via last.json.
  • Run: bun run eval:prompts-dispatch (local Ollama by default; EVAL_MODEL / EVAL_OLLAMA_URL overrides).

Results (smoke run, local qwen3:0.6b)

• 3/9 routed to the correct specialist; noDiy 9/9; fast 9/9; eval exits 0 within its allowances.
• Gate calibration note: routes: { miss: 6 } is intentionally loose for the 0.6B smoke model (observed 3/9); README says to tighten (e.g. miss: 1) when running a capable model. The drop gate against a stored capable-model baseline is the regression protection.

Verify

• bun scripts/eval-prompts-dispatch.ts → exit 0
• tsc --noEmit → clean (also run by pre-commit)
• oxfmt --check + oxlint on changed files → clean

Not in scope

• Jev (LLM-as-judge) checks need the @corbits/system-one gateway; code checks only for now.
• Trajectory scorers (ranSteps / @corbits/evals/workflow) are the CL-10452 follow-up.

[corbits-triage]

Triage: Pass

No changes needed from you. A maintainer will review this pull request.

[TheGreatAxios]

Review · Request changes

Wires @corbits/evals and a dispatch-routing prompt eval. The route gate can pass when routing is wrong, and the drop gate does not stick.

Findings

• scripts/eval-prompts-dispatch.ts:147 — routes is a case-insensitive substring over the whole reply, and expect.decoys is parsed and never checked. The reply format names every specialist, so a reply that is only the roster passes routes for every case. A reply of reviewer plus a refusal passes the coder case. Score the chosen route as its own line, and fail when a decoy is the chosen route.

• scripts/eval-prompts-dispatch.ts:185 — allow.routes is { miss: 6, drop: 0 } and last.json is gitignored. The library skips drops when that file is missing, and evaluate writes save before throwing. A clean tree never applies drop: 0. After a 9/9 baseline, a 4/9 run fails once and then exits 0 because the overwritten baseline is inside the miss budget. Keep the baseline that drop is measured against from being replaced by the failing run.

• scripts/eval-prompts-dispatch.ts:153 — noDiy scans the whole reply, including the required second-line brief. coder plus "I will just patch the login flow and add a test." passes. shakespeare plus "I'll write the architecture doc" fails. Limit the DIY scan to the route line, or stop treating brief verbs as DIY.

• scripts/eval-prompts-dispatch.ts:179 — under({ ms: 120_000 }) never aborts the run. timeout is not passed to evaluate, so a hung Ollama chat never becomes a fast miss and never unblocks the process. Pass a timeout that matches the budget.

[TheGreatAxios]

Review · Request changes

Dispatch routing is scored on the first line, and a stored pass that drops fails the run.

Findings

• scripts/eval-prompts-dispatch.ts:227 — routes allows 6 misses. A miss that is not a drop of a stored pass still returns 0 (:241). The baseline is copied only when routes.missed is 0, so a case that is not already stored as a pass can stay wrong, and a sibling that did pass on that run is never stored. A later flip of that sibling is not a drop. scripts/eval-prompts-dispatch.test.ts:281 locks the within-budget miss, not this hole.
• scripts/eval-prompts-dispatch.ts:188 — noDiy scores only routeLine, which is a specialist id. The DIY phrase list cannot fail a case routes would pass.

[TheGreatAxios]

Review · Approve

Dispatch routing fails an unbaselined route miss, stores a new pass next to a stored miss, and no longer runs the dead DIY check.

No findings.