Skip to content

fix(pilot): do not pair a seed-0 artifact with pooled multi-seed scores - #1137

Merged
Birfy merged 2 commits into
mainfrom
fix/pilot-multi-seed-artifact
Nov 17, 2026
Merged

Birfy merged 2 commits into
mainfrom
fix/pilot-multi-seed-artifact

Conversation

@Birfy

@Birfy Birfy commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

Summary

A multi-seed pilot run used to report seed 0's artifact, final_reward, and hash next to test scores pooled across every seed. That made it look like one candidate earned a number it did not.

This keeps the pooled interval as the method-level decision, but a single artifact is attached only when every seed produced the same one. When artifacts differ, run.per_seed carries each seed's hash, reward, and test gain, and the top-level reported_seed / artifact_hash / final_reward stay empty so the pooled scores cannot be pinned on one candidate. Default single-seed behavior is unchanged.

This is the seed-consistency fix from the FlashRSI design review (item 5). Item 4 (silent reflect=agent default) is intentionally untouched. The portable snapshot work (item 3) is a separate PR.

Tests

  • tests/test_pilot.py: existing two-seed pooling still holds when artifacts agree; a new test stubs different per-seed artifacts and fails if the report pairs pooled scores with seed 0 (or any single seed). Helper _shared_artifact is pinned for the agree / disagree cases. Single-seed reports stay the same.

Checklist

  • Changelog fragment will be added once this PR has a number
  • No new dependencies
  • No Cursor Cloud Coding Sessions

A multi-seed pilot still pools test scores for the method-level
interval, but it no longer reports seed 0's artifact, reward, or hash
as the run's result. A single artifact is attached only when every seed
produced it; otherwise the report summarizes per seed and the pooled
numbers describe the method, not one candidate. Default single-seed
behavior is unchanged.
@Birfy

Birfy commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

Self-review (Birfy)

Tip SHA: de2a97b
Ran:

  • python3 .github/scripts/risk_gate.py --repo ThinkFlowLab/agentdescent --pr 1137 → T2 (209 lines / 3 files)
  • pytest -q tests/test_pilot.py tests/test_changelog_fragments.py at this tip → 59 passed (14 pilot + changelog fragments)
  • python -m tools.changelog_fragments check → 64 changelog fragment(s) OK
  • python -m tools.gen_api_docs --check → docs/api.md is up to date
  • reproduced the bug on unmodified main 71fe47b: a two-seed stub with seed-0 reward 0.1 and seed-1 reward 0.9 reported final_reward=0.1 next to pooled n=24 and the note "the reported artifact is seed 0; the scores pool every seed"
  • the new test test_pooled_scores_are_not_paired_with_the_first_seed_artifact at this tip → passed; the report leaves reported_seed / artifact_hash / final_reward empty when hashes differ, keeps per-seed rows, and still pools n=24
    Not run:
  • Full offline suite locally — CI is running the 3.9–3.13 matrix at this tip
  • mkdocs build --strict — no docs page touched
  • Ark real-model run — no model path changed
    Findings:
  • none
    Verdict: PASS

@Birfy
Birfy enabled auto-merge (squash) November 16, 2026 23:51
@Birfy
Birfy merged commit d4b66cf into main Nov 17, 2026
10 checks passed
@Birfy
Birfy deleted the fix/pilot-multi-seed-artifact branch November 17, 2026 00:22
Birfy added a commit that referenced this pull request Nov 17, 2026
## Summary

This is item 3 from the FlashRSI design review: a portable snapshot on
the existing FileTree and ledger, not a second store and not a new
algorithm×harness×benchmark layer.

A snapshot is the complete candidate (editable files plus frozen
context) plus the lineage the ledger already has: `id` is
`{artifact_id}@v{version}`, `parent` is the previous version, `root` is
`{artifact_id}@v1`. Algorithms still edit only the declared scope;
evaluation and restore use the whole tree.

New surface:

- Library: `ArtifactSnapshot`, `evaluate_snapshot`,
`evaluate_snapshot_spec`, `read_artifact_lineage`
- CLI: `agentdescent snapshot export|show|eval|apply`
- Docs: `docs/snapshot.md` (README was not rewritten)

`snapshot eval` scores a frozen snapshot on a destination spec's tasks
and scorer. The spec's `target` is ignored; its `evolve` knobs do not
start a search. Kind must match. Host selection stays the existing
agent/runner path on that destination spec. `kind: plugin` is still the
plugin tree, not “run this snapshot inside Claude Code.”

`evolve(test_frac=…)` defaults and the silent `reflect=agent` default
are unchanged.

This is a separate PR from the seed-consistency fix in #1137; the two do
not share code.

## Tests

- `tests/test_snapshot.py`: frozen files are in the snapshot even if
omitted from an editable-only state; JSON round-trip keeps lineage;
lineage is read from the existing ledger (v3/v2/v1, distinct commits);
readonly eval does not call `evolve`; dest-spec eval uses dest tasks and
rejects a kind mismatch; a plugin snapshot is the plugin tree.
- `tests/test_snapshot_cli.py`: `show` / `export` / `eval` / `apply` on
a snapshot file, with `evolve` patched to fail if search starts.

## Checklist

- [x] Changelog fragment will be added once this PR has a number
- [x] No new dependencies
- [x] No Cursor Cloud Coding Sessions
- [x] MCP Path A tools were left unchanged; the new verb is library +
CLI. MCP parity can be a follow-up if we want it on the default tool
list.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant