FIX: SequentialAttack ends UNDETERMINED when no child reached a verdict

[WatchTree-19]

Fixes #2658

attack_outcome_from_score in pyrit/executor/attack/core/attack_strategy.py states the contract in its own docstring:

This is the attack-side contract for undetermined scores, stated once so no attack invents its own. An undetermined score is neither achievement nor refutation, so it never reads as failure; an attack that ends on one ends undetermined.

The leaf attacks honour it. PromptSendingAttack._determine_outcome and MultiPromptSendingAttack._determine_outcome both carry UNDETERMINED through explicitly rather than letting it fall into the failure branch.

SequentialAttack._compute_outcome did not. Both aggregation paths ended in an unconditional return AttackOutcome.FAILURE, so any sequence that produced no SUCCESS and was not entirely ERROR came back FAILURE, whether or not a single child attack had refuted anything.

Behaviour on main, exercised through _perform_async with stubbed child attacks:

policy	children	outcome
first_success	UNDETERMINED, UNDETERMINED	failure
first_decisive	UNDETERMINED, UNDETERMINED	failure
strict_all	SUCCESS, UNDETERMINED	failure
exhaustive	UNDETERMINED, UNDETERMINED	failure
exhaustive	UNDETERMINED, ERROR	failure
last_result	UNDETERMINED	undetermined

LAST_RESULT was the only policy that preserved it, and only because it inherits the final child's outcome verbatim rather than aggregating.

That matters downstream: scenario_run_service.py computes succeeded = sum(result.outcome == AttackOutcome.SUCCESS ...), so everything that is not SUCCESS lands in one bucket and a sequence nobody could call is presented as a clean negative. It gets worse as scorers become more willing to abstain. A scorer with a calibrated abstain band returns undetermined on exactly the responses it is least sure about, and those are the ones a red-team report should not quietly record as safe.

The change

FAILURE is reached only when a child attack actually refuted the objective. A sequence that reached no verdict ends UNDETERMINED. The existing SUCCESS and ERROR rules are untouched.

if any(r.outcome is AttackOutcome.SUCCESS for r in results):
    return AttackOutcome.SUCCESS
if all(r.outcome is AttackOutcome.ERROR for r in results):
    return AttackOutcome.ERROR
if any(r.outcome is AttackOutcome.FAILURE for r in results):
    return AttackOutcome.FAILURE
return AttackOutcome.UNDETERMINED

and the same ordering for STRICT_ALL after its SUCCESS and ERROR checks. The four policy docstrings that said "else FAILURE" now say what the rule actually is.

Two existing test expectations change, and that is the part to look at

This is not a change nobody had written down. Two parametrised cases in TestOutcomeDerivation currently assert the behaviour above:

• EXHAUSTIVE, [UNDETERMINED, UNDETERMINED], expecting FAILURE
• STRICT_ALL, [SUCCESS, UNDETERMINED], expecting FAILURE

Both now expect UNDETERMINED. I have flipped them rather than worked around them because they are the two that contradict the contract attack_outcome_from_score says is stated once for everyone. If the intent was that compound attacks are exempt from that contract, then this PR is wrong and the helper's docstring is what needs changing instead. I would rather be told that than guess.

The other 39 tests in the file are untouched. In particular EXHAUSTIVE, [UNDETERMINED, FAILURE] -> FAILURE still passes and still guards the half that matters: a real refutation anywhere in the sequence still produces FAILURE.

Three cases are added, for the policies that had no undetermined coverage at all: FIRST_SUCCESS and FIRST_DECISIVE over two undetermined children, and EXHAUSTIVE over undetermined-plus-error.

Verification

All five new or changed cases were run against unmodified main and fail there, and pass with the change. The full file is 44 passed. tests/unit/executor/attack/compound and tests/unit/executor/attack/core together are 202 passed, and the 58 tests in tests/unit/scenario/scenarios/adaptive, which consume SequentialAttack through the dispatcher, pass unchanged. ruff at the pinned v0.16.6 reports both files already formatted with all checks passing.

The em-dashes in the diff are pre-existing in the policy docstrings and were left alone.

Developed with AI assistance; the commit carries a Co-authored-by trailer.