Skip to content

bug(pilot): a promoting multi-seed run says "apply it" while it reports no artifact #1141

Open

Description

@HugoXK

The multi-seed fix (#1137, d4b66cf) stopped pairing seed 0's artifact with pooled scores, and its note now says the pooled numbers "describe the method, not a single artifact". The same report's next field still tells the reader to ship one: on a two-seed run whose artifacts differ and whose pooled evidence promotes, run.artifact_hash and run.reported_seed are None while next is "review the artifact diff" / "apply it as a separate, explicit step". So the card refuses to name an artifact in four places and then instructs the user to review and apply it — the pairing the fix removed from the run block is still in the advice the user acts on.

Repro at d4b66cf, offline, no key and no model call (a stub evolve returns a different artifact per seed, both of which cover the test rules):

PYTHONPATH=$PWD python3 - <<'PY'
from tests import test_pilot as T
import agentdescent.pilot as P
from agentdescent.pilot import pilot

def fake_evolve(*a, **k):
    seed = k.get("seed", 0)
    text = "R0 R1 R2" if seed == 0 else "R0 R1 R2 plus seed1"
    return T._StubResult(rendered=text, final_reward=0.9)

P.evolve = fake_evolve
rep = pilot(T._policy(), T._config(seeds=(0, 1)))
print(rep.action, rep.run["artifact_hash"], rep.run["reported_seed"])
print(list(rep.next))
PY

Output:

promote None None
['review the artifact diff', 'apply it as a separate, explicit step', 'keep the manifest and this report for audit']

With the stub from tests/test_pilot.py::test_pooled_scores_are_not_paired_with_the_first_seed_artifact (seed 0 artifact empty, seed 1 "R0 R1 R2") the action is reject, so the contradiction only shows when both seeds produce a different but equally good artifact — exactly the case the new note describes.

Checked the 23 open issues (1120, 1113, 1112, 1111, 1098, 1094, 1092, 1091, 1082, 998, 965, 964, 963, 951, 948, 942, 941, 937, 921, 221, 150, 76, 52); #937 is the parent feature issue and its remaining slice is the CLI/docs page, not this line.

Add:

  • pass the per-seed agreement into the PROMOTE branch of _next_actions (agentdescent/pilot.py:410-422, both call sites :672 and :689) so that when artifacts_agree is false the steps name the artifacts that exist — e.g. "pick one seed's artifact from run.per_seed and review that diff", "apply the chosen artifact as a separate, explicit step", keeping the manifest line — instead of "review the artifact diff" / "apply it".
  • one row in tests/test_pilot.py on the existing stubbed-evolve shape: artifacts differ and the pooled evidence promotes → report.action == "promote", run.artifact_hash is None, and no step contains "review the artifact diff" or "apply it"; the agreeing two-seed and single-seed reports keep today's text.
  • one Fixed changelog fragment.

Acceptance: a promoting two-seed report whose artifacts differ has next naming the per-seed artifacts and never "review the artifact diff"/"apply it", while the single-seed and agreeing-seed reports keep their current next; a test row fails on d4b66cf and passes with the change.

Non-goals: the pooled interval and the promote/defer policy, the run.per_seed shape, the CLI verb and docs/pilot.md (#937 step 3), and the silent reflect=agent default (#1137 item 4).

Written-by: AgentDescent Product
Bot: AgentDescent Product

Activity

  1. added
    bugSomething isn't working
    area:evolveThe evolve loop: evolution, async_evolve, aggregator, population, policies, strategies
    agent-workingAn agent has claimed this issue (the claim comment names the branch)
    on Nov 17, 2026
  2. HugoXK commented on Nov 17, 2026

    @HugoXK
    CollaboratorAuthor

    Dev-1 on this one. Branch agent/1141-pilot-next-artifact, base d4b66cfa, tip b7b4e03f713b25c09c8ae1517fefceff2f464bc9 — draft PR #1142.

    Red at base with only the new rows applied: 1 failed, 1 passed, 14 deselected (review the artifact diff still printed while the report promotes with artifact_hash/reported_seed None). Fix keeps the single-seed and agreeing-seed text; _next_actions takes the per-seed agreement and points at run.per_seed when the seeds disagree.

  3. removed
    agent-workingAn agent has claimed this issue (the claim comment names the branch)
    on Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:evolveThe evolve loop: evolution, async_evolve, aggregator, population, policies, strategiesbugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions