Skip to content

feat!: run evals with one runEvaluation call - #23

Closed
TheGreatAxios wants to merge 1 commit into
mainfrom
cl-11496-rename-the-eval-entrypoint-to-runevaluation
Closed

TheGreatAxios wants to merge 1 commit into
mainfrom
cl-11496-rename-the-eval-entrypoint-to-runevaluation

Conversation

@TheGreatAxios

Copy link
Copy Markdown
Contributor

Summary

  • Replace evaluate(name, options) with runEvaluation({ name, data, task, scorers }).
  • The spec is an Evaluation. The saved comparable result is an Experiment at format 2. Format 1 is rejected.
  • Report.checks and calibrate's public signature stay. The JSONL expect field stays for a later rename.

Verification

  • bun run check exited 0 (typecheck, oxlint, oxfmt, 1 unit test, 53 e2e tests).
  • Review of 0f62ede found no defects.

Fixes CL-11496

@linear-code

linear-code Bot commented Oct 26, 2026

Copy link
Copy Markdown

CL-11496

@corbits-triage

corbits-triage Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Triage: Pass

No changes needed from you. A maintainer will review this pull request.

@TheGreatAxios TheGreatAxios left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review · Approve

Replaces the eval entrypoint with one runEvaluation call (task, data, scorers).

Findings

No findings.

src/evaluate.ts is gone. In-repo call sites, examples, and the README use runEvaluation. The remaining evaluate import is @corbits/system-one in src/jev.ts, not the old entry. Passing a previous Report as compare is a type error (Report.cases is not an Experiment). bun test e2e src exited 0 (54 pass) and bun run typecheck exited 0.

@TheGreatAxios

Copy link
Copy Markdown
Contributor Author

Superseded by the 0.2.0 release: #32

Do not merge this pull request separately. The release branch cherry-picks the feature stack (except the 0.1.3 bump on #22) and sets @corbits/evals to 0.2.0 with @corbits/system-one ^0.4.0.

@TheGreatAxios

Copy link
Copy Markdown
Contributor Author

Superseded by the 0.2.0 release: #32

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant