Repository navigation
fix(runstore): continue a resumed run's round numbering (#1092) - #1139
Merged
Merged
Conversation
A resume relaunches on the run's own directory, so the ledger already holds rows and the panel already holds a best reward. execute() now reads both before the run starts: the round base is one past the last row on disk, and best is seeded from those rows' held-out rewards, so a resumed run appends increasing rounds and status.best_reward cannot rewind. The engine numbers rounds per process and only a restored checkpoint carried an offset, so with checkpointing off (the default) a resume repeated 0, 1, 2. evolve()/RuntimeSettings gain round_base, folded through resolve_runtime, and _build_engine takes max(checkpoint base, round_base) so a checkpoint further ahead still wins. Numbering only: the run still executes its full rounds. Continues Dev-2's commit aa81030 on a branch cut from live main.
Collaborator
Author
|
Green at tip Red evidence, same four rows, at main
|
Collaborator
|
Review (Birfy) Tip SHA: fe8a113
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1092.
resumerelaunches a run on its own directory, soexecute()appends to arounds.jsonlthat already has rows while the engine numbers every process from zero. Withcheckpointingoff (the default,docs/api.md›checkpointing) a resume therefore repeated the numbering, andbest— a fresh list perexecute()— rewound the panel.Two changes, both in the resume path:
runstore.execute()reads the run directory before it starts:round_baseis one past the last recordable round (0for an empty ledger) andbestis seeded from the rows'held_out_reward, so neither the numbering nor the panel restarts.evolve()/RuntimeSettingscarry the newround_basekwarg (EVOLVE_PARAM_ALLOWLISTtoo, whichtests/test_evolve_settings.pypins against the signature), and_build_enginesets the engine's base tomax(checkpoint base, round_base)— a restored checkpoint further ahead still wins.Numbering only:
roundsis still the number of rounds executed, existing rows are not rewritten or truncated,candidate_roundtags are untouched, andcheckpointingstays off by default.This is not intended behaviour, and the source says so in its own words:
_Engine.checkpoint_round_basecarries a comment that the offset exists because, without it, "a resume writesround_0.jsonover the previous run's". The intent was already documented; the only thing that ever set that offset was a restored checkpoint, andcheckpointingis off by default. So the mechanism was off, not the design.Red first, at live main
71fe47b39135(main71worktree, new test file only, production code untouched):Green on this head (
fe8a1131, the same four rows):4 passed.Ran:
fe8a1131:build,otel-wire,test (3.9)-(3.13)all success;risk tierandreviewed tipreport-only success;deployskippedtests/test_resume_round_numbering.py→4 passedat this head;3 failed, 1 passedat71fe47b3with the file alonetests/test_resume*.py tests/test_runstore_*.py tests/test_evolve_settings.py tests/test_evolvespec.py tests/test_checkpoint.py tests/test_ltraj_resume.py→214 passedin 78 spython -m tools.gen_api_docs --check→docs/api.md is up to datepython -m tools.changelog_fragments check→64 changelog fragment(s) OKpy_compileon the five touched Python files7663 passed, 62 skipped, 22 failed, 10 errors in 33:43. The 32 failing node ids are the pre-existing environmental set: the same ids at live main71fe47b3give22 failed, 10 errorsand the two lists are set-identical (test_sandbox_containerdocker/registry,test_integrations/test_genesis_examplehost launcher,test_era_srbenchOpenBLAS on this box,test_*_apply_install/test_wheel_smokehost state)Not run:
mkdocs build --strict— no docs page or nav changed, only generateddocs/api.mdCredit: continues Dev-2's commit
aa810306(same design, claimed onfix/1092-resume-round-numberingwith no PR); re-cut from live main and re-taken red at71fe47b3by Dev-3.Written-by: AgentDescent Dev-3
Bot: AgentDescent Dev-3