Skip to content

fix(eval): explain a non-positive gain as no improvement, not noise - #1147

Merged
Birfy merged 1 commit into
mainfrom
agent/1091-negative-gain-note
Nov 17, 2026
Merged

Birfy merged 1 commit into
mainfrom
agent/1091-negative-gain-note

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Fixes #1091.

What changed

_noise_refusal_note (trust_evidence.py:335) promised three causes in its docstring, but distinguishable (stats_eval.py) is false for four; gain <= 0.0 had no branch, so a regression fell through to whichever other cause was true (3 noisy repeats at gain −0.300 printed a ± 0.200 band; one repeat blamed the repeat count).

The note now branches first on noise.gain <= 0.0: <head> (the candidate did not improve on the baseline: the gain must be positive), with no second print of the gain. The docstring lists the four causes in code order. Every branch keeps not distinguishable from noise; distinguishable, the verdict strings and gain_over_noise are unchanged, as is the acceptance rule.

Three files, +51/−5: agentdescent/trust_evidence.py; tests/test_trust_evidence_noise.py (negative gain, exact gain == 0.0, and a +8.327e-16 control that must stay in the floor branch because the guard is <= 0.0, not a tolerance); changelog.d/1091-negative-gain-note.md.

Verification

Tip SHA: 866325e96e6e4213be6d3c7e160d92248a35a025

Red first, clean main 71fe47b3 with only the new rows: 2 failed, 9 passed — both failures assert 'did not improve on the baseline' in '… gain does not exceed the noise floor …' (test_a_regression_is_named_as_no_improvement_not_as_noise, test_an_exact_zero_gain_takes_the_no_improvement_branch).

Ran:

  • pytest -q tests/test_trust_evidence_noise.py → 11 passed
  • pytest -q tests/test_noise_calibration.py tests/test_diagnose.py → 33 passed
  • pytest -q on the three files together → 44 passed
  • python -m tools.changelog_fragments check → 64 changelog fragment(s) OK
  • python -m tools.gen_api_docs --check → docs/api.md is up to date

Not run:

  • full pytest -q → not achievable on this host (22 failed, 10 errors, all environment: docker sandbox, apply_noop OSError, wheel smoke, genesis, era_srbench, skill-library installs, shadowing cwd); the failing node-id sets are identical at base 71fe47b3. CI at the tip is the authoritative suite.
  • mkdocs build --strict — no docs page or mkdocs.yml touched.
  • Real-model run — no model path changed.

@HugoXK HugoXK added bug Something isn't working area:evolve The evolve loop: evolution, async_evolve, aggregator, population, policies, strategies agent-pr Pull request opened by an agent labels Nov 17, 2026
@Birfy

Birfy commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Review (Birfy)

Tip SHA: 866325e
Ran:

  • .github/scripts/risk_gate.py --pr 1147 → T2 (56 lines / 3 files)
  • pytest -q tests/test_trust_evidence_noise.py tests/test_noise_calibration.py tests/test_diagnose.py → 44 passed at this tip
  • the two new rows against main (e42d400) → test_a_regression_is_named_as_no_improvement_not_as_noise and test_an_exact_zero_gain_takes_the_no_improvement_branch failed (detail still blamed the noise floor); test_a_positive_float_noise_gain_stays_in_the_floor_branch passed, so a +8.327e-16 gain stays in the floor branch
  • python -m tools.changelog_fragments check → 64 changelog fragment(s) OK
  • python -m tools.gen_api_docs --check → docs/api.md is up to date
  • CI at this tip → required checks green (build, test 3.9–3.13); otel-wire SUCCESS; deploy skipped; risk tier and reviewed tip are report-only SUCCESS
    Not run:
  • Full offline suite locally — the CI test matrix is the line
  • Ark real-model run — no model path changed
    Findings:
  • none
    Verdict: PASS

@Birfy
Birfy merged commit 36d6af3 into main Nov 17, 2026
10 checks passed
@Birfy
Birfy deleted the agent/1091-negative-gain-note branch November 17, 2026 10:25
HugoXK added a commit that referenced this pull request Nov 18, 2026
Forward-merge after #1138/#1140/#1139/#1147/#1148/#1151/#1152 landed.
#1148 rewrites agentdescent/pilot.py, the file this branch patches; the
auto-merge keeps both the per-seed _next_actions branch and the
notes/strategy card addition, and the focused set is green at the
merge tree.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-pr Pull request opened by an agent area:evolve The evolve loop: evolution, async_evolve, aggregator, population, policies, strategies bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(eval): a regression is explained as a noise-floor problem — the note has no branch for gain <= 0

2 participants