Skip to content

fix(runstore): continue a resumed run's round numbering (#1092) - #1139

Merged
Birfy merged 1 commit into
mainfrom
agent/1092-resume-round-base
Nov 17, 2026
Merged

Birfy merged 1 commit into
mainfrom
agent/1092-resume-round-base

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1092.

resume relaunches a run on its own directory, so execute() appends to a rounds.jsonl that already has rows while the engine numbers every process from zero. With checkpointing off (the default, docs/api.md › checkpointing) a resume therefore repeated the numbering, and best — a fresh list per execute() — rewound the panel.

Two changes, both in the resume path:

  • runstore.execute() reads the run directory before it starts: round_base is one past the last recordable round (0 for an empty ledger) and best is seeded from the rows' held_out_reward, so neither the numbering nor the panel restarts.
  • evolve() / RuntimeSettings carry the new round_base kwarg (EVOLVE_PARAM_ALLOWLIST too, which tests/test_evolve_settings.py pins against the signature), and _build_engine sets the engine's base to max(checkpoint base, round_base) — a restored checkpoint further ahead still wins.

Numbering only: rounds is still the number of rounds executed, existing rows are not rewritten or truncated, candidate_round tags are untouched, and checkpointing stays off by default.

This is not intended behaviour, and the source says so in its own words: _Engine.checkpoint_round_base carries a comment that the offset exists because, without it, "a resume writes round_0.json over the previous run's". The intent was already documented; the only thing that ever set that offset was a restored checkpoint, and checkpointing is off by default. So the mechanism was off, not the design.

Red first, at live main 71fe47b39135 (main71 worktree, new test file only, production code untouched):

>       assert rows == [0, 1, 2, 3, 4, 5], f"resume renumbered the rounds: {rows}"
E       AssertionError: resume renumbered the rounds: [0, 1, 2, 0, 1, 2]
E       assert [0, 1, 2, 0, 1, 2] == [0, 1, 2, 3, 4, 5]
E         At index 3 diff: 0 != 3
1 failed, 3 deselected in 1.52s

>       assert seen["round_base"] == 3, (
E       AssertionError: execute() told the engine to start at None, not at one past the ledger's last round
E       assert None == 3
1 failed, 3 deselected in 0.80s

3 failed, 1 passed in 2.62s

Green on this head (fe8a1131, the same four rows): 4 passed.

Ran:

  • CI at this head fe8a1131: build, otel-wire, test (3.9)-(3.13) all success; risk tier and reviewed tip report-only success; deploy skipped
  • tests/test_resume_round_numbering.py → 4 passed at this head; 3 failed, 1 passed at 71fe47b3 with the file alone
  • tests/test_resume*.py tests/test_runstore_*.py tests/test_evolve_settings.py tests/test_evolvespec.py tests/test_checkpoint.py tests/test_ltraj_resume.py → 214 passed in 78 s
  • python -m tools.gen_api_docs --check → docs/api.md is up to date
  • python -m tools.changelog_fragments check → 64 changelog fragment(s) OK
  • py_compile on the five touched Python files
  • full offline suite on this head: 7663 passed, 62 skipped, 22 failed, 10 errors in 33:43. The 32 failing node ids are the pre-existing environmental set: the same ids at live main 71fe47b3 give 22 failed, 10 errors and the two lists are set-identical (test_sandbox_container docker/registry, test_integrations / test_genesis_example host launcher, test_era_srbench OpenBLAS on this box, test_*_apply_install/test_wheel_smoke host state)

Not run:

  • mkdocs build --strict — no docs page or nav changed, only generated docs/api.md
  • Ark real-model run — the row is deterministic offline

Credit: continues Dev-2's commit aa810306 (same design, claimed on fix/1092-resume-round-numbering with no PR); re-cut from live main and re-taken red at 71fe47b3 by Dev-3.

Written-by: AgentDescent Dev-3
Bot: AgentDescent Dev-3

@HugoXK HugoXK added agent-pr Pull request opened by an agent area:runstore Run store, ledger, checkpoint, resume and replay bug Something isn't working labels Nov 17, 2026
A resume relaunches on the run's own directory, so the ledger already holds
rows and the panel already holds a best reward. execute() now reads both
before the run starts: the round base is one past the last row on disk, and
best is seeded from those rows' held-out rewards, so a resumed run appends
increasing rounds and status.best_reward cannot rewind.

The engine numbers rounds per process and only a restored checkpoint carried
an offset, so with checkpointing off (the default) a resume repeated 0, 1, 2.
evolve()/RuntimeSettings gain round_base, folded through resolve_runtime, and
_build_engine takes max(checkpoint base, round_base) so a checkpoint further
ahead still wins. Numbering only: the run still executes its full rounds.

Continues Dev-2's commit aa81030 on a branch cut from live main.
@HugoXK
HugoXK requested a review from Birfy November 17, 2026 00:44
@HugoXK

HugoXK commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Green at tip fe8a1131f650a689565b39e6af9ad087f45595d9, and the head has not moved since this PR opened: build, otel-wire, test (3.9)–(3.13) all success, deploy skipped, both report-only rows success. Reviewing this sha.

Red evidence, same four rows, at main 71fe47b3 in a clean tree with only the new test file present:

  • assert rows == [0, 1, 2, 3, 4, 5] → AssertionError: resume renumbered the rounds: [0, 1, 2, 0, 1, 2], At index 3 diff: 0 != 3
  • assert seen["round_base"] == 3 → AssertionError: execute() told the engine to start at None, not at one past the ledger's last round
  • 3 failed, 1 passed; the same four rows on this branch → 4 passed.

@github-actions github-actions Bot added the risk:T3 Risk tier T3 (core loop, release, process; needs Danyang's click), set by the controller label Nov 17, 2026
@Birfy

Birfy commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Review (Birfy)

Tip SHA: fe8a113
Ran:

  • .github/scripts/risk_gate.py --pr 1139 → T3 (paths agentdescent/async_evolve.py, evolution.py, runstore.py; 210 lines / 6 files)
  • pytest -q tests/test_resume_round_numbering.py → 4 passed at this tip
  • the same four rows on main (fcebbdf) with only the new test file present → 3 failed, 1 passed (empty ledger still starts at 0). Red matches the PR: resume renumbered [0, 1, 2, 0, 1, 2]; execute() passed round_base=None; evolve(round_base=) is an unexpected keyword on main
  • confirmed checkpoint_round_base is fully renamed (no leftover references) and save_checkpoint uses the already-offset index
  • python -m tools.changelog_fragments check → 64 changelog fragment(s) OK
  • python -m tools.gen_api_docs --check → docs/api.md is up to date
  • CI at this tip → required checks green (build, test 3.9–3.13); otel-wire SUCCESS; deploy skipped; risk tier and reviewed tip are report-only SUCCESS
    Not run:
  • Full offline suite locally — the CI test matrix is the line
  • Ark real-model run — no model path changed
    Findings:
  • none
    Verdict: PASS

@Birfy
Birfy merged commit e42d400 into main Nov 17, 2026
10 checks passed
@Birfy
Birfy deleted the agent/1092-resume-round-base branch November 17, 2026 04:24
HugoXK added a commit that referenced this pull request Nov 18, 2026
Forward-merge after #1138/#1140/#1139/#1147/#1148/#1151/#1152 landed.
#1148 rewrites agentdescent/pilot.py, the file this branch patches; the
auto-merge keeps both the per-seed _next_actions branch and the
notes/strategy card addition, and the focused set is green at the
merge tree.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-pr Pull request opened by an agent area:runstore Run store, ledger, checkpoint, resume and replay bug Something isn't working risk:T3 Risk tier T3 (core loop, release, process; needs Danyang's click), set by the controller

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(runstore): resume renumbers rounds from 0 and rewinds status.round and best_reward

2 participants