Repository navigation
Run BC-Bench's Copilot CLI call through the core CopilotHarness - #1014
Merged
Sun Haoran (haoranpb) merged 2 commits intoNov 6, 2026
Merged
Sun Haoran (haoranpb) merged 2 commits into
Sun Haoran (haoranpb) merged 2 commits into
Conversation
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are reported for the change moving prompt rendering from agent harnesses into the CLI.
No open issues 🎉
What changed in this PR
Moves prompt rendering from agent harnesses into the CLI, separating benchmark policy from harness execution. This continues #1013’s work toward harnesses that consume prepared agent input.
Changes:
- Adds
AgentTaskto carry a dataset entry and its rendered prompt through evaluation requests. - Moves Copilot and Claude prompt rendering into CLI commands and updates all BC-Bench harnesses to accept the new task structure.
- Updates fixtures and tests to verify prompt forwarding, including the
al_mcpoption.
| File | Description |
|---|---|
tests/test_review_runners.py |
Uses concrete entries and checks wrapped harness tasks. |
tests/test_pr_review_agent.py |
Wraps PR Review test input in AgentTask. |
tests/test_evaluate_pipeline.py |
Updates the test pipeline’s harness type. |
tests/test_dataquery_evaluation.py |
Supplies a prompt in evaluation contexts. |
tests/test_copilot_agent.py |
Passes prompts directly instead of mocking rendering. |
tests/test_cli_commands.py |
Checks prompt forwarding and rendering options. |
tests/test_claude_code_agent.py |
Supplies a rendered prompt directly to Claude. |
tests/test_bcal_agent_provider.py |
Wraps BCal test input with its prompt. |
tests/conftest.py |
Updates shared harness and context fixtures. |
src/bcbench/types.py |
Defines AgentTask and forwards context prompts. |
src/bcbench/evaluate/base.py |
Updates pipeline signatures for wrapped tasks. |
src/bcbench/commands/run.py |
Prepares prompts before invoking harnesses. |
src/bcbench/commands/evaluate.py |
Populates evaluation prompts and updates mock types. |
src/bcbench/agent/pr_review/agent.py |
Extracts the entry from the wrapped task. |
src/bcbench/agent/copilot/agent.py |
Consumes the supplied prompt without rendering it. |
src/bcbench/agent/claude/agent.py |
Consumes the supplied prompt without rendering it. |
src/bcbench/agent/bcal/agent.py |
Uses the supplied prompt for invocation and logging. |
Review effort: Balanced
Sun Haoran (haoranpb)
force-pushed
the
haoranpb/prompt-out-of-harness
branch
from
November 5, 2026 13:07
e42ca22 to
db14622
Compare
Sun Haoran (haoranpb)
changed the title
The core CopilotHarness gets a required name (keeping StrEnum names intact in identity) and an optional experiment configuration reported with results and timeouts. BC-Bench's CopilotHarness still prepares each task but delegates the CLI invocation and timeout handling to it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sun Haoran (haoranpb)
changed the title
Sun Haoran (haoranpb)
force-pushed
the
haoranpb/prompt-out-of-harness
branch
from
November 5, 2026 14:11
db14622 to
1e375c2
Compare
Rename the application AgentHarness registry to HarnessName and the core Harness protocol to AgentHarness. Update implementations, consumers, tests and contributor guidance without changing runtime behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Contributor
Code coverage overviewLanguages: Python Python / code-coverage/pytest
View coverage of the 10 most impacted files.
ℹ️ Calculations represent line coverage. Results may change as calculations are updated. |
Wenjie Fan (gggdttt)
approved these changes
Nov 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1013.
HarnessNameand the execution protocol toAgentHarness.No runtime behavior change. Validation: 264 focused tests, Ruff, workspace type checks, and pre-commit passed.