[TheGreatAxios]

Summary

• Add a corbits-eval bin. corbits-eval run <file> loads an evaluation spec and calls runEvaluation.
• Exit 0 when the run returns, 1 when it throws EvalFailure, and 2 when the file is missing or has no spec.
• The bin name is corbits-eval only.

Verification

• bun run check exited 0 (typecheck, lint, format, unit, and e2e: 72 passed).
• Review of e9ed49d found no defects.

Stacked on #27.

Fixes CL-11497

[linear-code]

CL-11497

[corbits-triage]

Triage: Pass

No changes needed from you. A maintainer will review this pull request.

Stacked on #27: only this pull request's own changes were checked.

Waiting on a maintainer to approve: approve, request review.

[TheGreatAxios]

Review · Request changes

Adds the corbits-eval CLI. The bin, command discovery, and the four existing exit-code tests are wired. --format json is not machine-readable, and a file that is not a spec exits 1 instead of 2.

Findings

• src/cli/run.ts:21 — runEvaluation(spec) prints the human summary before the JSON body. bun src/cli/index.ts run e2e/fixtures/pass.ts --format json exits 0, but stdout starts with pass: 1/1 cases passed, so JSON.parse throws Unexpected identifier "pass". A gate failure (--json on e2e/fixtures/fail.ts, exit 1) has the same prefix. The command never sets quiet: true and does not look at c.format. README’s --format json / --json contract is not something jq can read.

• src/evalFile.ts:18 — isSpec only requires a string name plus the keys data and task. export const evaluation = { name: "bad", data: 1, task: "nope" } is accepted, then bun src/cli/index.ts run <that file> exits 1 with {"code":"UNKNOWN","message":"cases.map is not a function..."} instead of exit 2 and “no evaluation spec”. A non-spec is reported as a failed gate.

• src/cli/run.ts:27 — an empty data: [] exits 1 with code: UNKNOWN and evaluation "empty" has no data. A scorer of "is it fine?" with no judge exits 1 with string and ask checks need judge.invoke. README reserves exit 1 for a failed gate (EVAL_FAILED). These config errors never ran, but they take the same exit code.

package.json bin corbits-eval points at ./dist/cli/index.js. bun run build, bun run typecheck, and bun run test:e2e exited 0 (72 pass). Those e2e cases never assert stdout or a bad spec shape. The Node failure Bun is not defined is the existing scorer path, not this CLI branch.

[TheGreatAxios]

Review · Follow-up

Commit 5f7e328 is clean.

--format json and --json print only JSON (quiet: true). A pass exits 0 and a gate failure exits 1 with EVAL_FAILED. A non-spec export exits 2 with “no evaluation spec”. Empty data and a string scorer with no judge exit 2 with EVAL_CONFIG. bun test e2e/cli.e2e.test.ts exited 0 (9 pass). A deleted probe also checked JSONL data, a traced task, an agent send task, and a throwing task.

[TheGreatAxios]

Superseded by the 0.2.0 release: #32

Do not merge this pull request separately. The release branch cherry-picks the feature stack (except the 0.1.3 bump on #22) and sets @corbits/evals to 0.2.0 with @corbits/system-one ^0.4.0.

[TheGreatAxios]

Superseded by the 0.2.0 release: #32