Skip to content

fix(judge): write the final message once in the trial, failure and pair states - #181

Merged
aitianshi merged 2 commits into
ThinkFlowLab:mainfrom
opPO333:fix/judge-final-message-once
Nov 17, 2026
Merged

aitianshi merged 2 commits into
ThinkFlowLab:mainfrom
opPO333:fix/judge-final-message-once

Conversation

@opPO333

@opPO333 opPO333 commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Closes #177

What

The trial, failure and pair states wrote the final message twice. _transcript wrote the step it came from as an [agent] line. build then appended the same text under its heading. build now asks transcript_of for a one-line marker in place of that step: [agent] [the final message; its text is under [final message] below], and [final message A] or [final message B] in the pair state. An earlier step with the same text stays as it is, because the step is found by its position among the agent steps, by the rule last_agent_message uses. No observation moves between kept and omitted, because the relevance rule still reads the real text of each message. The final and control states are unchanged. transcript_of with its present arguments, the call control_state makes, returns the same text as before. Only the deprecated question better_of_two reads the pair state. Nothing has asked it since v2.

Touches

backend

Verified

  • NULLIUS_HOME=$(mktemp -d) python3 -m unittest discover -s app/tests -t app -p 'test_judge.py': 42 tests, OK.
  • The new class FinalMessageOnce (6 tests), run on the test commit before the fix: 4 tests fail, 6 failures with subtests. After the fix: OK.
  • The reproduction snippet of the issue prints trial 1, failure 1, final 1, pair 2.
  • Backend suite, python3 app/tests/parallel.py, on a copy of the branch on a Linux file system: 2,053 tests in 119 modules, OK.
  • A three-way merge of this branch with the fix for Judge: the failure state leaves the [verifier] section empty when the verifier wrote nothing #178: the judge tests run 46 tests, OK.
  • Route suite: not run.
Acceptance Test Result
trial and failure write the step of the last agent message once, under [final message] test_each_kind_holds_the_final_message_once_per_trajectory, test_an_earlier_step_with_the_same_text_stays_and_the_last_is_the_marker pass
A one-line marker stands in the place of that message, at the same position among the steps test_the_marker_stands_where_the_final_message_stood, test_an_agent_step_after_the_final_message_stays_after_the_marker pass
In the pair state the final message of each side is written once test_each_kind_holds_the_final_message_once_per_trajectory, test_the_marker_stands_where_the_final_message_stood pass
The final and control states are unchanged test_each_kind_holds_the_final_message_once_per_trajectory (final), test_the_control_state_keeps_the_last_message_in_its_transcript pass
A test builds each kind from a fixture and counts the occurrences test_each_kind_holds_the_final_message_once_per_trajectory pass
The pull request says whether VERSION changes Risk, below stated

Risk

VERSION in app/nullius/judge/questions.py stays judgement.questions.v2. Its rule bumps it when a question is reworded. This change rewords no question. Records stored before this change were asked on the previous state text. That text held the final message twice. The spread in app/nullius/judge/floors.json was measured on that text too. If the marker lands on the wrong step, that step's text is missing from the state. A revert of the fix commit restores the previous state text.

The trial, failure and pair states write the final message twice: once as
the last [agent] line of the transcript and again under its heading. These
tests build each kind from the judge fixtures and from made-up trajectories,
and expect the step of the final message to be a one-line marker at its
place among the steps.
…ir states

The transcript wrote the step that last_agent_message came from as an
[agent] line, and build appended the same text under [final message], so
the text reached the judge twice and a long one pushed the end of the
transcript out of a truncated state. build now asks transcript_of for a
marker that names the heading. The step is found by its position, so an
earlier step with the same text stays. transcript_of without the heading,
and so the control state, is unchanged.
@opPO333
opPO333 marked this pull request as draft November 17, 2026 02:10
@opPO333
opPO333 marked this pull request as ready for review November 17, 2026 13:35
@aitianshi
aitianshi merged commit 8839bbe into ThinkFlowLab:main Nov 17, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Judge: the trial, failure and pair states hold the final message twice

2 participants