-
Notifications
You must be signed in to change notification settings - Fork 24
docs: nest Scorers, Self-improvement, and Benchmarks under Measure and improve (GROW-6131) #721
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
48be311
63639fb
8c689ae
aa8a0ae
59e5b4f
b6f0ba1
5634fff
6c35f8d
5987f6c
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| --- | ||
| title: Configuring Scorers | ||
| description: >- | ||
| Configure a Scorer to have an LLM judge classify completed factory runs | ||
| against criteria you write. | ||
| sidebar: | ||
| label: "Scorers" | ||
| --- | ||
|
|
||
| A **Scorer** uses an LLM judge to classify completed runs against criteria you write — for example, "did the agent run the tests before opening a PR?" It assigns a label, not a numeric grade, so keep each Scorer focused on one question its failures can point back to. Create Scorers on the factory dashboard's **Scorers** page, which also holds each Scorer's results. For Scorers defined as files in a factory definition, see the [`scorers/<name>/scorer.md` syntax](/factories/factory-as-code/#scorersnamescorermd) and the two scorers in [`02-sdlc-issue-to-pr`](https://github.com/warpdotdev/warp-factory-examples/tree/main/examples/02-sdlc-issue-to-pr) in the [warp-factory-examples](https://github.com/warpdotdev/warp-factory-examples) repository. | ||
|
|
||
| Configure these fields: | ||
|
|
||
| * **Agent(s) to evaluate** - The agents this Scorer applies to. Select at least one. | ||
| * **Judge instructions** - The criteria the judge checks for. | ||
| * **Judge model** - The model that acts as the judge. | ||
| * **Classifications** - The labels the judge can assign, each with a score. | ||
| * **Pass threshold** - The score a run needs to pass. | ||
| * **Sample rate** - The share of the selected agents' completed runs to evaluate. | ||
|
|
||
| {/* VISUAL: The Scorer create/edit form (judge instructions, judge model, classifications, pass threshold, sample rate). */} | ||
|
|
||
| While the sample rate is above 0, scoring happens automatically: shortly after a sampled run completes, the judge evaluates it and records a classification, a score, and its reasoning. To stop automatic scoring, set the sample rate to 0. | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. this is a bit confusing, maybe we can just give an example like, "if the sample rate is 10%, that means 10% of runs of this XYZ agent will be judged using this scorer"? that might be a bit easier to grok |
||
|
|
||
| You can also score any single run on demand, which is useful for testing new judge instructions before raising the sample rate. Scoring a run again replaces its previous result from that Scorer. | ||
|
|
||
| Changing **Pass threshold** updates how past scores display as pass or fail; the recorded results don't change. | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. maybe we can remove this sentence, im not too sure what it means |
||
|
|
||
| ## Related pages | ||
|
|
||
| * [Measure and improve a factory](/factories/measure-and-improve/) - Read dashboard metrics and run a practical improvement loop. | ||
| * [Configuring and reviewing Self-improvement](/factories/measure-and-improve/self-improvement/) - Turn repeated Scorer failures into follow-up pull requests. | ||
| * [Benchmarking factory agent configurations](/factories/benchmarks/) - Compare model and runner configurations on the same tasks. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| --- | ||
| title: Configuring and reviewing Self-improvement | ||
| description: >- | ||
| Turn on Self-improvement for a Scorer to turn its repeated failures into | ||
| follow-up pull requests you review. | ||
| sidebar: | ||
| label: "Self-improvement" | ||
| --- | ||
|
|
||
| {/* VISUAL: The Self-improvement pull request list, or a Benchmarks suite run -- this section is text-only today. */} | ||
|
|
||
| Turn on **Self-improvement** for each Scorer whose failures you want investigated automatically. Self-improvement groups related failures and files follow-up tasks as ordinary agent runs. A follow-up run can propose changes to application code. It can also improve the factory itself: when you manage your factory as [definitions as code](/factories/factory-as-code/), its prompts, skills, and configuration are version-controlled files, so a follow-up run can open a pull request against the factory definition the same way it would against application code. Nothing is adopted without your review. | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can we explain a bit more about when self-improvement runs are triggered - the codebase should be able to tell you that it's something like you need a minimum of 25 failures of a certain scorer for self-improvement to run (which is definitely not very intuitive). @vkodithala just rolled out the ability for you to kickoff self-improvement runs adhoc, but also configure the thresholds of which self-improvement runs (# of failures required), etc. so we should document that as well! |
||
|
|
||
| The factory dashboard's **Self-improvement** page lists the pull requests these follow-up runs open. Each pull request includes a **Regressions addressed** section that links the failing runs and Scorer results behind it, so you can trace the change back to its evidence. | ||
|
|
||
| ## Related pages | ||
|
|
||
| * [Measure and improve a factory](/factories/measure-and-improve/) - Read dashboard metrics and run a practical improvement loop. | ||
| * [Configuring Scorers](/factories/measure-and-improve/scorers/) - Define the criteria whose failures Self-improvement investigates. | ||
| * [Benchmarking factory agent configurations](/factories/benchmarks/) - Compare model and runner configurations on the same tasks. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder if we can explain what a scorer is a bit better in this description, maybe using language like "eval" -- I think the word eval helps folks understand scorers
Similarly, in the first paragraph as well