fix(evaluations): apply pass_rate_threshold to EvalRunResult.passed

[donei003]

Problem

EvalRunResult.passed was true only when no row failed, errored, or was pending. It ignored pass_rate_threshold. A run with 28 passed, 1 failed, and 1 error rows (93%) returned passed=False when each criterion set pass_rate_threshold=0.9.

Change

• Add _run_passed in module.py.
• If every criterion sets pass_rate_threshold, the run passes when no row is pending and passed_rows / total_rows is at least the highest threshold. Errored rows count as not passed.
• If any criterion has no pass_rate_threshold, the strict rule stays. Any failed, errored, or pending row fails the run.
• An empty run never passes.

Limit

The summary endpoint returns row counts for the whole run. It does not return counts for each criterion. The check uses the highest threshold on the whole-run pass rate. A per-criterion check needs per-criterion counts from the API.

Test

Add a parametrized test for _run_passed. The full client suite, ruff, and mypy pass.

🤖 Generated with Claude Code

---

---

Note

Overview
EvalRunResult.passed now respects pass_rate_threshold on evaluation criteria instead of requiring zero failed rows.

A new _run_passed helper drives the gate: when every criterion sets a pass rate, the run passes once nothing is pending and passed_rows / total_rows meets the highest threshold (failed and error rows lower the rate). If any criterion omits a pass rate, behavior stays strict—any failed, errored, or pending row fails the run. Empty runs never pass under pass-rate mode.

Parametrized unit tests cover mixed thresholds, partial pass-rate configs, and edge cases. Package versions in uv.lock bump to 0.2.4 (and related packages).

^{Reviewed by Cursor Bugbot for commit ea171a9. Bugbot is set up for automated code reviews on this repo. Configure here.}

[devin-ai-integration]

🔍 Devin Review: 1 flag

Not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)