Repository navigation
chore(release): 0.2.0 - #32
Merged
Merged
Conversation
JSONL rows that still say expect or expected are rejected.
jev() keeps thresholds and escalation only. judgeModel, exported from the workflow entry, forces a single step whose input is the case payload.
defineWorkflow stamps a lone step with { from: "trigger.payload" } even when the author omits input. judgeModel treated that stamp as a custom input and never ran the step.
An empty step list is not an observed empty tool path, so ranSteps([]) fails unless a watched run actually used no tools.
JSON output sets quiet so the human summary does not prefix stdout. A module that is not a runnable spec is rejected at load time. Config errors that never start a run exit 2 instead of 1.
State format 2 experiments, expectedOutput JSONL rows, and ranSteps([]) on an empty step list.
Depend on @corbits/system-one ^0.4.0. Leave the lockfile unchanged because 0.4.0 is not published yet.
This was referenced Oct 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #31
Release branch for
@corbits/evals0.2.0. Do not merge and do not npm publish until@corbits/system-one0.4.0 is on npm.Version
package.jsonis0.2.0(not 0.1.3).@corbits/system-one^0.4.0, not a git URL and not 0.3.x.bun installcannot resolve^0.4.0yet (No version matching "^0.4.0"; npm currently lists through 0.3.2). The lockfile is intentionally unchanged so it is not committed while it still resolves 0.3.0. Update the lockfile after 0.4.0 publishes, then runbun run check.Stack
Cherry-picked onto
origin/main(0.1.2), skipping #22:feat!: run evals with one runEvaluation callfeat!: name the case ground truth expectedOutputfeat!: run the judge as one workflow stepandfix: accept a one-step judge workflow's default inputfeat: add string includes and prebuilt judge scorersfeat: record agent steps from stream(),fix: isolate aborted turns and name denied tools, andfix: hold an aborted turn on the raw send rejectionfeat: add the corbits-eval CLIandfix: print only json and exit 2 when an eval does not runfeat: rename the jev constructor to systemOnedocs: describe the current eval APIanddocs: correct how prebuilt rubrics are runREADME exit code 2 covers an evaluation that did not run (empty
data, missingjudge.invoke), not only a missing file. The prebuilt section saysjudgeModeldoes not run those rubrics.PRs #22-#30 are superseded by this branch and must not be merged separately.
Check
bun run checkdid not run. Install failed waiting on the@corbits/system-one0.4.0 publish.