Repository navigation
fix(judge): write the final message once in the trial, failure and pair states - #181
Merged
aitianshi merged 2 commits intoNov 17, 2026
Merged
Conversation
The trial, failure and pair states write the final message twice: once as the last [agent] line of the transcript and again under its heading. These tests build each kind from the judge fixtures and from made-up trajectories, and expect the step of the final message to be a one-line marker at its place among the steps.
…ir states The transcript wrote the step that last_agent_message came from as an [agent] line, and build appended the same text under [final message], so the text reached the judge twice and a long one pushed the end of the transcript out of a truncated state. build now asks transcript_of for a marker that names the heading. The step is found by its position, so an earlier step with the same text stays. transcript_of without the heading, and so the control state, is unchanged.
opPO333
marked this pull request as draft
November 17, 2026 02:10
opPO333
marked this pull request as ready for review
November 17, 2026 13:35
aitianshi
approved these changes
Nov 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #177
What
The
trial,failureandpairstates wrote the final message twice._transcriptwrote the step it came from as an[agent]line.buildthen appended the same text under its heading.buildnow askstranscript_offor a one-line marker in place of that step:[agent] [the final message; its text is under [final message] below], and[final message A]or[final message B]in thepairstate. An earlier step with the same text stays as it is, because the step is found by its position among the agent steps, by the rulelast_agent_messageuses. No observation moves between kept and omitted, because the relevance rule still reads the real text of each message. Thefinalandcontrolstates are unchanged.transcript_ofwith its present arguments, the callcontrol_statemakes, returns the same text as before. Only the deprecated questionbetter_of_tworeads thepairstate. Nothing has asked it since v2.Touches
backend
Verified
NULLIUS_HOME=$(mktemp -d) python3 -m unittest discover -s app/tests -t app -p 'test_judge.py': 42 tests, OK.FinalMessageOnce(6 tests), run on the test commit before the fix: 4 tests fail, 6 failures with subtests. After the fix: OK.trial 1,failure 1,final 1,pair 2.python3 app/tests/parallel.py, on a copy of the branch on a Linux file system: 2,053 tests in 119 modules, OK.trialandfailurewrite the step of the last agent message once, under[final message]test_each_kind_holds_the_final_message_once_per_trajectory,test_an_earlier_step_with_the_same_text_stays_and_the_last_is_the_markertest_the_marker_stands_where_the_final_message_stood,test_an_agent_step_after_the_final_message_stays_after_the_markerpairstate the final message of each side is written oncetest_each_kind_holds_the_final_message_once_per_trajectory,test_the_marker_stands_where_the_final_message_stoodfinalandcontrolstates are unchangedtest_each_kind_holds_the_final_message_once_per_trajectory(final),test_the_control_state_keeps_the_last_message_in_its_transcripttest_each_kind_holds_the_final_message_once_per_trajectoryVERSIONchangesRisk
VERSIONinapp/nullius/judge/questions.pystaysjudgement.questions.v2. Its rule bumps it when a question is reworded. This change rewords no question. Records stored before this change were asked on the previous state text. That text held the final message twice. The spread inapp/nullius/judge/floors.jsonwas measured on that text too. If the marker lands on the wrong step, that step's text is missing from the state. A revert of the fix commit restores the previous state text.