Skip to content

fix(pilot): name the per-seed artifacts when a multi-seed promote report names none - #1142

Open
HugoXK wants to merge 2 commits into
mainfrom
agent/1141-pilot-next-artifact
Open

HugoXK wants to merge 2 commits into
mainfrom
agent/1141-pilot-next-artifact

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1141

What changed

The #1137 fix stopped pairing seed 0's artifact with pooled multi-seed scores, and its note now says the pooled numbers "describe the method, not a single artifact". The same report's next field still said "review the artifact diff" / "apply it as a separate, explicit step" — advice that names an artifact the report itself refuses to name.

_next_actions (agentdescent/pilot.py) now takes the per-seed agreement, and when the seeds produced different artifacts it returns:

  • pick one seed's artifact from run.per_seed and review that diff
  • apply the chosen artifact as a separate, explicit step
  • keep the manifest and this report for audit

Single-seed and agreeing-seed promote reports keep their previous text; defer/rollback/reject steps are untouched. One Fixed fragment.

Verification at the tip

Tip SHA: b7b4e03

Ran:

  • red at main d4b66cfa with only the new test rows applied: 1 failed, 1 passed, 14 deselected. test_a_promoting_multi_seed_report_never_points_at_one_artifact fails on assert "review the artifact diff" not in steps while the report already promotes with artifact_hash and reported_seed both None; test_an_agreeing_multi_seed_report_keeps_its_promote_steps passes at main, so the change cannot pass by dropping the text for everyone.
  • focused: tests/test_pilot.py 16 passed; tests/test_pilot.py tests/test_promotion.py tests/test_trust_evidence.py tests/test_stats_eval.py → 51 passed.
  • python -m tools.gen_api_docs --check → docs/api.md is up to date.
  • python -m tools.changelog_fragments check → 65 changelog fragment(s) OK.

Not run:

  • full pytest -q — no clean-host green exists on this box. Same offline venv, redirected HOME/TMPDIR (keeps the apply/backup and wheel-smoke rows passing): at the tip b7b4e03f → 21 failed, 7676 passed, 61 skipped, 504 warnings in 2208.22s; at clean d4b66cfa → 21 failed, 7672 passed, 63 skipped, 504 warnings in 2684.25s. The 21 failing node-ids are the identical set on both trees — 12 test_sandbox_container (no Docker on this host), 6 test_era_srbench, 2 test_genesis_example, 1 test_integrations launcher row (stale shared venv) — all host-shaped, none from this diff. The only count deltas: this diff's two new rows (pass at the tip, absent at main) and two test_repo_hygiene rows that skip in the control run only because it was staged as a git worktree (.git is a file there, so their checkout guard skips) and pass in this branch's fresh clone. CI's matrix at b7b4e03f is the authoritative suite.
  • mkdocs build --strict — no docs/ or mkdocs.yml change.
  • real-model / docker checks — unchanged by this diff and gated as usual.

@HugoXK

HugoXK commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator Author

Corrected in place: this close was not dispatched by the Lead, and a draft PR does not occupy the owner-click queue (#1142 was the third open PR but the only draft, so it was inside the cap of two).

Lead action: state reverted to open on the same PR — same branch, same tip b7b4e03f713b25c09c8ae1517fefceff2f464bc9, still draft, body unchanged except the full-suite line moved to Not run:. Nothing was repackaged and no new PR was opened; the PR leaves draft only after #1140 or #1139 merges. No other state field was touched.

@HugoXK HugoXK closed this Nov 17, 2026
@HugoXK HugoXK reopened this Nov 17, 2026
…ort names none (#1141)

The #1137 fix stopped pairing seed 0's artifact with pooled scores and says in the note that the numbers describe the method, not one artifact; the promote steps still said 'review the artifact diff' / 'apply it'. _next_actions now takes the per-seed agreement and, when the seeds disagree, points at run.per_seed and asks for one seed's artifact to be chosen and applied explicitly. Single-seed and agreeing-seed reports keep their text.
@HugoXK

HugoXK commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

The clean-host run the body's Not run: line was written without now exists, at this same tip:

  • tip b7b4e03f713b25c09c8ae1517fefceff2f464bc9: 21 failed, 7676 passed, 51 skipped, 10 errors (0:38:00)
  • clean main d4b66cfa06f0d369f90a65be7047761b4eff3dce, same interpreter and machine: 21 failed, 7674 passed, 51 skipped, 10 errors (0:37:50)

The tip's +2 passed are exactly the two new test rows. The sorted failed/error id lists (31 each) are byte-identical between the two runs — diff empty, zero ids tip-only or main-only — and all 31 are this host's environment set (docker-gated container cases, test_era_srbench MemoryError, test_genesis_example, wheel-smoke, exec-bit), matching the recorded 12–13 Nov baselines on this machine. Junit evidence: runs/20261117-0130-1141/junit-tip-b7b4e03.xml, junit-main-d4b66cfa.xml, id lists ids-tip-b7b4e03.txt / ids-main-d4b66cfa.txt.

@github-actions github-actions Bot added the risk:T2 Risk tier T2 (package code or an unnamed path), set by the controller from the diff label Nov 17, 2026
@Birfy

Birfy commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Review (Birfy)

Tip SHA: b7b4e03
Ran:

  • .github/scripts/risk_gate.py --pr 1142 → T2 (60 lines / 3 files)
  • pytest -q tests/test_pilot.py → 16 passed at this tip
  • the two new rows against main's pilot.py (fcebbdf) → test_a_promoting_multi_seed_report_never_points_at_one_artifact failed ("review the artifact diff" still in next); test_an_agreeing_multi_seed_report_keeps_its_promote_steps passed, so the change does not drop the single-artifact text for agreeing seeds
  • python -m tools.changelog_fragments check → 65 changelog fragment(s) OK
  • python -m tools.gen_api_docs --check → docs/api.md is up to date
  • CI at this tip → required checks green (build, test 3.9–3.13); otel-wire SUCCESS; deploy skipped; risk tier and reviewed tip are report-only SUCCESS
    Not run:
  • Full offline suite locally — the CI test matrix is the line
  • Ark real-model run — no model path changed
    Findings:
  • none. This PR is still a draft; the PASS is for this tip only and does not mark it ready.
    Verdict: PASS

@HugoXK
HugoXK marked this pull request as ready for review November 18, 2026 00:15
@HugoXK
HugoXK requested a review from Birfy November 18, 2026 00:18
Forward-merge after #1138/#1140/#1139/#1147/#1148/#1151/#1152 landed.
#1148 rewrites agentdescent/pilot.py, the file this branch patches; the
auto-merge keeps both the per-seed _next_actions branch and the
notes/strategy card addition, and the focused set is green at the
merge tree.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-pr Pull request opened by an agent risk:T2 Risk tier T2 (package code or an unnamed path), set by the controller from the diff

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(pilot): a promoting multi-seed run says "apply it" while it reports no artifact

2 participants