Repository navigation
Re-running individual failed test jobs fails because shared infrastructure has been torn down #590
Description
Maintainers and contributors run the Process-PSModule.yml workflow on every pull request. The workflow consists of a BeforeAll-ModuleLocal job that provisions shared GitHub infrastructure (organization and user repositories named Test-{OS}-{TokenType}-{RunID}), a fan-out matrix of Test-ModuleLocal jobs (one per test file × OS × auth case), and an AfterAll-ModuleLocal job that tears everything down. The setup and teardown were introduced in #541 so that the matrix could share repositories instead of each leaf job creating its own.
Request
When a small number of leaf test jobs in the Test-ModuleLocal matrix fail because of intermittent issues (rate limits, transient API errors, GitHub-side flakiness), the only available recovery path on the GitHub Actions UI is Re-run failed jobs or Re-run all jobs. Neither path produces a working rerun for this workflow:
What happens
- The matrix completes with a mix of successful and failed leaf jobs (the screenshot on the originating discussion shows
Test-Releases,Test-Repositories, andTest-Variablesred while ~25 sibling jobs are green). AfterAll-ModuleLocalis configured withif: ... && always()and runs as soon asTest-ModuleLocalfinishes, regardless of the matrix outcome. It deletes everyTest-{OS}-{TokenType}-{RunID}*repository for the run.- The user clicks Re-run failed jobs.
- GitHub Actions reruns only the failed leaf jobs in
Test-ModuleLocal. It does not rerunBeforeAll-ModuleLocalbecause that job succeeded the first time, and skipped/successfulneedsare not re-executed by Re-run failed jobs. - The rerun leaf jobs immediately fail in their per-context
BeforeAll, where they callGet-GitHubRepository -Name "Test-{OS}-{TokenType}-{RunID}"against a repository that no longer exists.
What is expected
A user who wants to retry transiently failing leaf jobs can do so without having to rerun every successful sibling job. The shared infrastructure that the leaf jobs depend on must be present (or re-created) before the rerun executes, so the targeted retry succeeds when the underlying flake has cleared.
Reproduction steps
- Open a pull request against
mainso thatProcess-PSModule.ymlruns end-to-end. - Wait for the workflow to finish with at least one failed leaf job in the
Test-ModuleLocalmatrix and theAfterAll-ModuleLocaljob completed (green). - From the workflow run page, click Re-run failed jobs.
- Observe that every reran leaf job fails in its
BeforeAllblock while resolving the shared repository — the repository was deleted byAfterAll-ModuleLocalafter step 2.
Regression
This behavior was introduced by #541. Before that change, every leaf test job created and deleted its own repositories using [guid]::NewGuid(), so reruns were self-contained — each leaf job rebuilt the infrastructure it needed. Consolidating to shared run-scoped infrastructure removed that property.
Workaround
The only working recovery today is Re-run all jobs, which re-executes every successful leaf job. For a workflow where a single matrix leg can take over two hours (Test-Apps (macOS) ran 1h 40m in the linked screenshot), this wastes substantial CI time and consumes additional GitHub API quota that the shared-repo design in #541 was specifically introduced to conserve.
Acceptance criteria
- A user can trigger a retry of a specific failed leaf job (or a subset of failed leaf jobs) and have it succeed when the underlying transient cause has cleared, without rerunning every successful sibling.
- The shared infrastructure the leaf jobs depend on is guaranteed to exist when those jobs execute, regardless of whether they are part of the original run or a partial rerun.
- Successful leaf jobs from the original run are not re-executed unnecessarily on a partial retry.
- Concurrent workflow runs continue to be isolated from each other (the run-scoped naming guarantee from #541 is preserved).
- API call budget for a partial retry is bounded — a retry of one leaf job does not cost more than the leaf job itself plus whatever minimum setup is unavoidable.
Activity
MariusStorhaug commentedon May 2, 2026
Additional analysis and suggestions for discussion
Posting this separately from the issue body so the issue stays focused on the problem statement. Suggestions below are starting points for discussion, not a recommended solution.
Constraints derived from reading the harness
GITHUB_RUN_IDis stable across reruns;GITHUB_RUN_ATTEMPTincrements per attempt.tests/BeforeAll.ps1is already idempotent for a givenGITHUB_RUN_ID— it pre-cleansTest-{OS}-{TokenType}-{RunID}*before creating, so invoking it multiple times for the same run is safe.Get-GitHubRepository), with one exception —Repositories.Tests.ps1still owns its own create/delete lifecycle.PSModule/Process-PSModule, not in this repo. Some options require changes there; some can be done entirely withinPSModule/GitHub.needs. Re-run all jobs does, but defeats the purpose of partial retries. There is no built-in "rerun this leaf and its setup chain" option.Options to discuss
AfterAll-ModuleLocal— run only on final attempt; rely on a scheduled sweeper for staleTest-*-{old-RunID}reposBeforeAll(idempotent get-or-create)BeforeAll-ModuleLocalto re-execute on every attemptworkflow_dispatchretry workflow taking a run ID + test file names, which provisions infra under the sameGITHUB_RUN_IDand reruns the named files viapwsh tests/...GITHUB_RUN_ATTEMPTand rebuild every attemptBeforeAllactually re-runs on partial retries.Initial lean
Options (2) and (6) are the only ones implementable entirely inside
PSModule/GitHub. Option (1) is the cleanest architecturally if Process-PSModule is open to it. The shared-repo design from #541 was specifically to conserve API quota, so any option that adds per-leaf API calls (2 standalone) needs to quantify the additional cost against that budget.Open questions for the discussion:
PSModule/Process-PSModulein scope, or should the fix live entirely in this repo?Repositories.Tests.ps1(which still owns its own lifecycle) follow whatever pattern is chosen, for consistency?MariusStorhaug commentedon May 2, 2026
Two prerequisite issues have been filed that must land before the persistent shared-repo model can be implemented safely:
#591 — Replace hardcoded external dependency on
ryanoasis/nerd-fontsinReleases.Tests.ps1. Currently one test callsGet-GitHubRelease -Owner ryanoasis -Repository nerd-fontsto exercise the pipeline path. This external resource can disappear or change at any time, is unrelated to anything this module owns, and consumes additional API quota. It should use assets already uploaded to the shared test repository within the same test context.#592 — Scope release tag names with
$idand addAfterAllcleanup of releases and git tags inReleases.Tests.ps1. Tags are currently hardcoded (v1.0–v1.4) and never cleaned up. On a persistent shared repository, sequential runs collide on the firstNew-GitHubRelease -Tag 'v1.0'call; concurrent runs race on the same tag. Both issues are latent today and guaranteed failures once the persistent-repo model is in place.Both issues are purely in
tests/Releases.Tests.ps1and have no upstream or module-code dependencies.MariusStorhaug commentedon May 2, 2026
A related issue has been filed: #593 — Seed test fixture files (including a
workflow_dispatch-enabled workflow YAML) into the shared test repository to enable testing of the full Workflow function surface and to re-home artifact tests away from their current dependency on the livePSModule/GitHubrepository andGITHUB_RUN_ID.This is independent of the persistent shared-repo work in this issue but benefits from it: once the shared repository is persistent and consistently seeded, workflow and artifact tests can be run reliably both in CI and locally without the current bootstrapping dependencies.
⚙️ [Maintenance]: Partial test reruns no longer fail when shared infr…