Skip to content

fix(judge): fill the failure state's verifier section when the verifier wrote nothing - #182

Merged
aitianshi merged 2 commits into
ThinkFlowLab:mainfrom
opPO333:fix/judge-empty-verifier-section
Nov 17, 2026
Merged

aitianshi merged 2 commits into
ThinkFlowLab:mainfrom
opPO333:fix/judge-empty-verifier-section

Conversation

@opPO333

@opPO333 opPO333 commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Closes #178

What

When none of the five files that verifier_text reads exists, the failure state ended with the heading [verifier] and no text below it. The judge could not tell whether the verifier ran and wrote nothing or never ran. With no verifier file, the section now says nothing recorded. control_state uses the same words. When trial is given and records neither a reward nor a verifier phase (Trial.reward and Trial.dur_verifier are both None), the section says did not run: the trial records no reward and no verifier phase. A trial stopped at its time limit before its verifier phase is such a case. A present verifier file is written as before. The control state is unchanged.

Touches

backend

Verified

  • NULLIUS_HOME=$(mktemp -d) python3 -m unittest discover -s app/tests -t app -p 'test_judge.py': 40 tests, OK.
  • The new class VerifierSection (4 tests), run on the test commit before the fix: 3 tests fail, 5 failures with subtests. The fourth test pins the unchanged case of a present verifier file and passes before and after the fix. After the fix: OK.
  • The reproduction snippet of the issue now ends with '[verifier]\nnothing recorded'.
  • Backend suite, python3 app/tests/parallel.py, on a copy of the branch on a Linux file system: 2,051 tests in 119 modules, OK.
  • A three-way merge of this branch with the fix for Judge: the trial, failure and pair states hold the final message twice #177: the judge tests run 46 tests, OK.
  • Route suite: not run.
Acceptance Test Result
The failure state never ends with an empty [verifier] section; with no verifier file it says nothing was recorded test_no_verifier_file_and_no_trial_is_nothing_recorded, test_a_trial_with_a_reward_or_a_verifier_phase_is_nothing_recorded pass
A trial with no reward and no verifier phase: the section says the verifier did not run test_a_trial_with_no_reward_and_no_verifier_phase_says_the_verifier_did_not_run pass
The section is unchanged when a verifier file exists test_a_verifier_file_is_the_section_whatever_the_trial_records pass
A test covers the three cases the four tests above pass
The pull request says whether VERSION changes Risk, below stated

Risk

VERSION in app/nullius/judge/questions.py stays judgement.questions.v2. Its rule bumps it when a question is reworded. This change rewords no question. Records stored before this change were asked on the previous state text. That text ended with an empty [verifier] section. The spread in app/nullius/judge/floors.json was measured on that text too. The sentence about a verifier that did not run rests on two fields of result.json. A trial whose verifier ran, but whose result records neither a reward nor a verifier phase, would read did not run. A revert of the fix commit restores the previous state text.

With no verifier file the failure state ends with an empty [verifier]
heading. These tests cover a missing file with no trial, with a trial that
records no reward and no verifier phase, and with a trial that records
either, and a present file, whose section stays as it is.
When none of the verifier's files exists, the [verifier] section was left
empty, so the judge could not tell whether the verifier ran and wrote
nothing or never ran. The section now says "nothing recorded", the wording
of the control state, and says the verifier did not run when the trial
records neither a reward nor a verifier phase. A present verifier file is
written as before.
@opPO333
opPO333 marked this pull request as draft November 17, 2026 02:10
@opPO333
opPO333 marked this pull request as ready for review November 17, 2026 13:35
@aitianshi
aitianshi merged commit 5cd922b into ThinkFlowLab:main Nov 17, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Judge: the failure state leaves the [verifier] section empty when the verifier wrote nothing

2 participants