fix(ci-diagnose): exclude Post cleanup steps from classify_failure's skipped-work count (#973)

[avrabe]

Closes #973.

What

classify_failure in tools/ci-diagnose/diagnose.sh counted every skipped step after the first failed step as evidence that the real work never ran, and returned setup-failed. GitHub's per-action Post … cleanup hooks are scheduled by the runner after the workflow's own steps and are routinely marked skipped once any earlier step has failed. That is why PR #970's job 105030664579 misclassified:

8  success  Run cargo-mutants
9  failure  Check surviving mutants     ← the real failure: 4 missed mutants
10 success  Upload mutants report
18 skipped  Post Run Swatinem/rust-cache@v2

The assertion step failed, the report was uploaded, and only the runner's own cleanup was skipped. classify_failure returned setup-failed, which is the misdiagnosis this classifier exists to prevent (REQ-316/REQ-317), reached the other way — the work did run, and its own check said the change was bad.

Fix

Exclude any step whose name starts with Post from skipped_after. Those are the runner's own cleanup hooks (GitHub schedules and names them; users cannot). A Post skip is not evidence the workflow's work never ran; only a non-Post skipped step is.

-      [.steps[$i + 1 :][]? | select(.conclusion == "skipped")] | length
+      [.steps[$i + 1 :][]?
+       | select(.conclusion == "skipped"
+                and ((.name // "") | startswith("Post ") | not))]
+      | length

Acceptance criteria (from #973)

• Excludes Post … steps from skipped_after. The startswith("Post ") guard on .name is the only change to the classifier.
• Adds the case from the issue to diagnose_test.sh and it classifies as job-failed. assertion failed, only Post cleanup skipped — reduced from PR fix(yaml): multi-line plain scalars were truncated at their first line on the rowan read path (REQ-363) #970's job 105030664579.
• The existing Kani case still classifies as setup-failed. The skipped work step there is Run cargo kani -p rivet-core, which is a workflow step and not a Post hook, so the existing case is unchanged. A new mixed case (setup failed with a Post skip still after it) also pins that a Post skip trailing a real skipped work step cannot silence the setup failure — the exclusion is safe.

Evidence

$ bash tools/ci-diagnose/diagnose_test.sh
…
classify_failure:
  ok   runner went away
  ok   genuine test failure
  ok   setup failed, work skipped
  ok   assertion failed, only Post cleanup skipped     ← new, from #973
  ok   setup failed with a Post skip still after it    ← new, mixed control
  ok   success is not a failure

ci-diagnose: all cases pass

41 cases green with the fix, and stashing the classifier change while keeping the tests reddens exactly the one new #973-derived case:

FAIL assertion failed, only Post cleanup skipped: expected 'job-failed', got 'setup-failed'
ci-diagnose: 1 case(s) FAILED

The rest of the file (classify_stall, normalize_runner_fetch, workflow-needs.py, the liveness cases) is untouched.

🤖 Generated with Claude Code

https://claude.ai/code/session_01T1QYjjHmT5DQU6EMuEVq8V

---

Generated by Claude Code

[github-actions]

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'Rivet Criterion Benchmarks'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.20.

Benchmark suite	Current: d302719	Previous: f16245a	Ratio
store_lookup/100	1941 ns/iter (± 4)	1606 ns/iter (± 10)	1.21
store_by_type/100	96 ns/iter (± 0)	76 ns/iter (± 0)	1.26
store_by_type/1000	96 ns/iter (± 0)	76 ns/iter (± 0)	1.26
store_by_type/10000	96 ns/iter (± 0)	76 ns/iter (± 0)	1.26
diff/1000	708535 ns/iter (± 3190)	581112 ns/iter (± 7534)	1.22
query/100	1117 ns/iter (± 18)	910 ns/iter (± 14)	1.23
query/1000	14112 ns/iter (± 25)	11672 ns/iter (± 131)	1.21

This comment was automatically generated by workflow using github-action-benchmark.

[avrabe]

Kani Proofs failed — not this PR's failure, standing down on it.

• Failing check: Kani Proofs (check_run 105437603255) on head d302719.
• Cause named in the job log: the model-checking/kani-github-action@v1 install step exits with ##[error]The version of Kani installed was different than the one requested, then ##[error]Process completed with exit code 1.. Install PropProof, Add PropProof to config, and Run Kani are all skipped after it. This is a setup failure inside the action itself; no proof body ran.
• Same failure on base main @ f16245a (my PR's base): CI run 35258650049 shows Kani Proofs: conclusion=failure while CI Gate: conclusion=success. Kani is not a needs of the gate. Every recent push to main has been the same shape — issue ci: Kani Proofs fails ~58% with exit 143 and has never passed on a push to main #839 (open) tracks that Kani has never passed on a push to main.
• My diff does not touch anything Kani-related. The only files changed in this PR are tools/ci-diagnose/diagnose.sh and tools/ci-diagnose/diagnose_test.sh, both shell. There is no path by which they can change how model-checking/kani-github-action@v1 installs the Kani release bundle.
• No fix exists to port. The underlying action-vs-verifier version mismatch is the subject of ci: Kani Proofs fails ~58% with exit 143 and has never passed on a push to main #839 and no fix PR is open for it. Making Kani robust is out of scope for a tools/ci-diagnose/ change.
• Not re-running. The rules allow one re-run to confirm reproducibility of an unrelated failure; base-branch reproduction on the same commit f16245a already satisfies that criterion, so re-running would just spend the allowance without adding evidence. If Kani flips green independently on a later push, no action needed.

Keeping the PR watched. If any other check on this PR reddens I will act on it; this one is documented and left as-is.

---

Generated by Claude Code

[avrabe]

Criterion regression alert — cannot be caused by this PR's diff, so no push from here.

• Diff scope: tools/ci-diagnose/diagnose.sh and tools/ci-diagnose/diagnose_test.sh, 41 insertions / 1 deletion. git diff origin/main..HEAD -- '*.rs' '*.toml' returns empty — no Rust source, no Cargo manifest, no lockfile touched. There is no code path by which this diff can change the timing of store_lookup, store_by_type, diff, or query in rivet-core.
• The bot compares d302719 (this PR) against f16245a (main). Both binaries link the identical rivet-core object files. Whatever +21% to +26% this table shows is between two Rust builds whose inputs are byte-identical — the delta lives outside the source.
• The shape of the table gives it away too. store_by_type/100, store_by_type/1000, and store_by_type/10000 all report exactly 96 vs 76 ns/iter — three inputs spanning two orders of magnitude produce the same timing. That is per-call overhead, not a hot loop; a 20 ns absolute drift there is well within self-hosted-runner variance (frequency scaling, cache warmth, sibling load), which is exactly what a threshold of 1.20 on a single sample will trip on. Two runs of the identical binary against main would very likely trip it too.
• No fix to port; no patch to widen this PR with. Widening a tools/ci-diagnose/*.sh PR to change how Criterion runs on CI would be exactly the "widening" the rules tell me not to do. The benchmark-baseline / noise-band problem lives in .github/workflows/benchmarks.yml and its baseline snapshot, and should be its own change if it needs one.

Keeping the PR watched. Everything else on head d302719 is either green (YAML Lint, Schema version bump, Detect changed areas) or skipped by the changed-areas filter (Format / Clippy / Test / Miri / etc., correctly, since no Rust files changed). Kani stood down separately above.

---

Generated by Claude Code