Repository navigation
fix(pilot): do not pair a seed-0 artifact with pooled multi-seed scores - #1137
Merged
Merged
Conversation
A multi-seed pilot still pools test scores for the method-level interval, but it no longer reports seed 0's artifact, reward, or hash as the run's result. A single artifact is attached only when every seed produced it; otherwise the report summarizes per seed and the pooled numbers describe the method, not one candidate. Default single-seed behavior is unchanged.
Collaborator
Author
|
Self-review (Birfy) Tip SHA: de2a97b
|
Birfy
enabled auto-merge (squash)
November 16, 2026 23:51
4 tasks done
Birfy
added a commit
that referenced
this pull request
Nov 17, 2026
## Summary
This is item 3 from the FlashRSI design review: a portable snapshot on
the existing FileTree and ledger, not a second store and not a new
algorithm×harness×benchmark layer.
A snapshot is the complete candidate (editable files plus frozen
context) plus the lineage the ledger already has: `id` is
`{artifact_id}@v{version}`, `parent` is the previous version, `root` is
`{artifact_id}@v1`. Algorithms still edit only the declared scope;
evaluation and restore use the whole tree.
New surface:
- Library: `ArtifactSnapshot`, `evaluate_snapshot`,
`evaluate_snapshot_spec`, `read_artifact_lineage`
- CLI: `agentdescent snapshot export|show|eval|apply`
- Docs: `docs/snapshot.md` (README was not rewritten)
`snapshot eval` scores a frozen snapshot on a destination spec's tasks
and scorer. The spec's `target` is ignored; its `evolve` knobs do not
start a search. Kind must match. Host selection stays the existing
agent/runner path on that destination spec. `kind: plugin` is still the
plugin tree, not “run this snapshot inside Claude Code.”
`evolve(test_frac=…)` defaults and the silent `reflect=agent` default
are unchanged.
This is a separate PR from the seed-consistency fix in #1137; the two do
not share code.
## Tests
- `tests/test_snapshot.py`: frozen files are in the snapshot even if
omitted from an editable-only state; JSON round-trip keeps lineage;
lineage is read from the existing ledger (v3/v2/v1, distinct commits);
readonly eval does not call `evolve`; dest-spec eval uses dest tasks and
rejects a kind mismatch; a plugin snapshot is the plugin tree.
- `tests/test_snapshot_cli.py`: `show` / `export` / `eval` / `apply` on
a snapshot file, with `evolve` patched to fail if search starts.
## Checklist
- [x] Changelog fragment will be added once this PR has a number
- [x] No new dependencies
- [x] No Cursor Cloud Coding Sessions
- [x] MCP Path A tools were left unchanged; the new verb is library +
CLI. MCP parity can be a follow-up if we want it on the default tool
list.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A multi-seed
pilotrun used to report seed 0's artifact,final_reward, and hash next to test scores pooled across every seed. That made it look like one candidate earned a number it did not.This keeps the pooled interval as the method-level decision, but a single artifact is attached only when every seed produced the same one. When artifacts differ,
run.per_seedcarries each seed's hash, reward, and test gain, and the top-levelreported_seed/artifact_hash/final_rewardstay empty so the pooled scores cannot be pinned on one candidate. Default single-seed behavior is unchanged.This is the seed-consistency fix from the FlashRSI design review (item 5). Item 4 (silent
reflect=agentdefault) is intentionally untouched. The portable snapshot work (item 3) is a separate PR.Tests
tests/test_pilot.py: existing two-seed pooling still holds when artifacts agree; a new test stubs different per-seed artifacts and fails if the report pairs pooled scores with seed 0 (or any single seed). Helper_shared_artifactis pinned for the agree / disagree cases. Single-seed reports stay the same.Checklist