Repository navigation
fix(judge): fill the failure state's verifier section when the verifier wrote nothing - #182
Merged
aitianshi merged 2 commits intoNov 17, 2026
Conversation
With no verifier file the failure state ends with an empty [verifier] heading. These tests cover a missing file with no trial, with a trial that records no reward and no verifier phase, and with a trial that records either, and a present file, whose section stays as it is.
When none of the verifier's files exists, the [verifier] section was left empty, so the judge could not tell whether the verifier ran and wrote nothing or never ran. The section now says "nothing recorded", the wording of the control state, and says the verifier did not run when the trial records neither a reward nor a verifier phase. A present verifier file is written as before.
opPO333
marked this pull request as draft
November 17, 2026 02:10
opPO333
marked this pull request as ready for review
November 17, 2026 13:35
aitianshi
approved these changes
Nov 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #178
What
When none of the five files that
verifier_textreads exists, thefailurestate ended with the heading[verifier]and no text below it. The judge could not tell whether the verifier ran and wrote nothing or never ran. With no verifier file, the section now saysnothing recorded.control_stateuses the same words. Whentrialis given and records neither a reward nor a verifier phase (Trial.rewardandTrial.dur_verifierare bothNone), the section saysdid not run: the trial records no reward and no verifier phase. A trial stopped at its time limit before its verifier phase is such a case. A present verifier file is written as before. Thecontrolstate is unchanged.Touches
backend
Verified
NULLIUS_HOME=$(mktemp -d) python3 -m unittest discover -s app/tests -t app -p 'test_judge.py': 40 tests, OK.VerifierSection(4 tests), run on the test commit before the fix: 3 tests fail, 5 failures with subtests. The fourth test pins the unchanged case of a present verifier file and passes before and after the fix. After the fix: OK.'[verifier]\nnothing recorded'.python3 app/tests/parallel.py, on a copy of the branch on a Linux file system: 2,051 tests in 119 modules, OK.failurestate never ends with an empty[verifier]section; with no verifier file it says nothing was recordedtest_no_verifier_file_and_no_trial_is_nothing_recorded,test_a_trial_with_a_reward_or_a_verifier_phase_is_nothing_recordedtest_a_trial_with_no_reward_and_no_verifier_phase_says_the_verifier_did_not_runtest_a_verifier_file_is_the_section_whatever_the_trial_recordsVERSIONchangesRisk
VERSIONinapp/nullius/judge/questions.pystaysjudgement.questions.v2. Its rule bumps it when a question is reworded. This change rewords no question. Records stored before this change were asked on the previous state text. That text ended with an empty[verifier]section. The spread inapp/nullius/judge/floors.jsonwas measured on that text too. The sentence about a verifier that did not run rests on two fields ofresult.json. A trial whose verifier ran, but whose result records neither a reward nor a verifier phase, would readdid not run. A revert of the fix commit restores the previous state text.