Repository navigation
Support current-directory output files in MT-bench - #3940
Open
j45856021-dev wants to merge 1 commit into
Open
j45856021-dev wants to merge 1 commit into
j45856021-dev wants to merge 1 commit into
Conversation
Handle an empty parent path consistently in API/local answer generation and single/pairwise judgment output. Preserve nested directory creation and add offline regression coverage for all four writers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
Using
--answer-file answers.jsonlfails after answer generation becauseos.path.dirname("answers.jsonl")is empty andos.makedirs("")raisesFileNotFoundError. The single and pairwise judgment writers have the same issue when called with a bare output filename.Use the current directory when there is no parent path. This changes only the four directory-creation calls; nested paths still create their parent directories, and JSONL output stays unchanged. Document the answer-file override.
The offline regression tests invoke all four real writer paths with API/model generation and CUDA transfer mocked. Before the fix, all four bare-filename cases fail; the four nested-directory controls pass. After the fix, all eight cases pass, including answer-file reorganization and saved result fields.
Checks
python -m unittest discover -s tests -p test_llm_judge_output_files.py -v: 4 tests passed, covering 8 writer/path combinations.format.sh --files .... Its repository-wide Pylint step fails with the same 886 diagnostics on both this change and the untouched base587d5cf; there are no added diagnostics.git diff --checkpasses. The new test file also parses with Python 3.8 syntax rules.Validation ran on Python 3.10. No model weights, remote completions, or GPU inference were used.