ops: diagnose and bound organization GitHub Actions queue starvation

Operational gap

Required current-head checks across ContextualWisdomLab/fast-mlsirm and the organization .github repository are remaining in queued state while obsolete heads accumulate. This blocks policy-compliant auto-merge even after CodeRabbit succeeds and review threads are resolved.

Representative evidence observed on 2026-08-03:

fast-mlsirm#410 current-head CI/Security/SAST runs queued after prior head jobs were cancelled;
fast-mlsirm#391, #396, and #401 exact-head CI/Security/SAST runs queued;
.github#704 exact-head CodeQL, Security, SAST, OSV, Scorecard, SBOM, Secret Scan, and Python Security runs queued;
.github#650, which repairs Rust coverage provisioning for fast-mlsirm, is itself waiting on queued required checks.

The repositories must not solve this by bypassing required checks or force-merging stale evidence.

Required diagnosis

Determine whether the queue is caused by hosted-runner concurrency, Actions billing/usage limits, organization policy, a reusable-workflow recursion, pending deployment/environment approval, or workflow-level concurrency keys.
Produce a machine-readable health snapshot containing repository, workflow, run id, event, head SHA, queue age, concurrency group, and whether the run is obsolete.
Distinguish a genuinely queued current-head run from a completed/cancelled job whose run-level status is stale or misleading.
Identify and cancel only obsolete runs; never cancel the sole current-head required evidence.
Add organization-level concurrency and close-event cancellation contracts where repository-local controls cannot operate before merge.
Alert on current-head required checks that remain queued beyond a declared SLO.
Preserve fail-closed merge policy: an unavailable runner is incomplete evidence, not success.

Acceptance

A deterministic script and scheduled workflow report queue health across allowlisted CWL repositories.
At most one pending/running instance exists per PR-number/workflow evidence lane, except explicitly documented matrix jobs.
Closed PRs and superseded heads do not retain expensive active jobs.
A current-head run can be traced from PR to workflow/job state without relying on contradictory run-level summaries.
The report identifies the external/manual action when billing, organization policy, or runner capacity cannot be fixed in code.
Tests, documentation, least-privilege permissions, and 100% coverage/docstrings are included.

Related work

fast-mlsirm#410: repository-local current-head concurrency and bounded review churn.
.github#704: hourly review-feedback repair scheduler.
.github#650: Rust coverage toolchain/offline dependency provisioning.

@jules

@jules Please implement this as a bounded organization-governance PR from current main. Start with TDD contract tests for run/job state normalization, current-head versus obsolete classification, queue-age SLOs, and fail-closed external-blocker reporting. Keep the first slice read-only: no cancellation or mutation until the diagnostic manifest proves exact repository/workflow/PR/head identity and tests cover contradictory run/job summaries. Use least-privilege Actions/PR read permissions, deterministic JSON plus accessible HTML, 100% coverage/docstrings, and normal reviewed PR publication only.

External-status check on 2026-08-03: GitHub's public status page reports Actions operational and the most recent documented Actions capacity incidents as resolved. Prioritize organization-specific causes first (billing/usage cap, concurrency saturation, environment approval, reusable-workflow recursion, or stale run-state normalization), while retaining an external-incident field in the health manifest. Official status: https://www.githubstatus.com/

Current queue-health evidence for .github#1013@ca6ba3ec313ba6551376d942f00d1739eb1e9176 on 2026-08-15:

CodeQL PR run 31884269793 remains queued.
Python Security run 31884269775 remains queued.
The same exact head already has successful OSV, Strix changed-path quality, SBOM, Scorecard, Security Scan, Secret Scan, and Semgrep runs.
No source failure is currently attributable to the two queued lanes; they remain incomplete evidence and are not treated as success.

A separate control-plane reproduction also shows review-router queue interference: scheduled mention sweep 31887686442 remained in progress while trusted interactive run 31888050967 was pending without jobs and then cancelled. That incident is tracked with terminal evidence in #814 and should not be conflated with the CodeQL/Python Security runner queue, but both currently delay exact-head integration.

Fresh OriginWeave queue-starvation evidence, refreshed 2026-08-17 KST:

Protected OriginWeave main remains exact 0841d2ab3d8b5e60a03c0a8e818cf438e2716829 after protected merge Use workflow token for scheduler reads #179.
PR Install target R package dependencies for OpenCode coverage #175 exact head 52e42370ce49ad6142fb19eb0211f43716f19803 and protected PR Use workflow token for scheduler reads #179 exact feature head 16a872cddd0e336adfe0a686a75d057c05d8c74a both ultimately cleared CI/MV3/SAST/Security without product-source churn; Use workflow token for scheduler reads #179 integrated to protected main. This remains evidence that the recurring unassigned-runner class is infrastructure/control-plane behavior rather than a leaf source defect.
PR docs: record aFIPC required workflow coverage gap #124 has advanced to current exact head 4561282c9c0cab5181b5d07d4f072f835826efc0 against protected main 0841d2ab3d8b5e60a03c0a8e818cf438e2716829. The previous e5ae33f7c7f73a0daa4daf9e18c00b6382b68d05 snapshot is historical only.
The latest docs: record aFIPC required workflow coverage gap #124 change is itself test-first resilience hardening: test commit 3eace839515c0268d62bce6714ec30974b65839e required successful HTTP 200 registry pages to reject retry_after_seconds; production commit 4561282c9c0cab5181b5d07d4f072f835826efc0 adds the narrow fail-closed validation. It does not convert any failed or queued GitHub evidence into success.
Native CI run 31975554347 is success on exact head 4561282...: Rust-contract job 95234397116 and production-coverage job 95234397086 both completed successfully.
The same exact head still has local scanners waiting without runner assignment: SAST run 31975554357, Semgrep job 95234379145; Security Scan run 31975554326, jobs 95234379359 (OSV), 95234379372 (Trivy FS), 95234379377 (dependency review), and 95234379465 (Scorecard). Fresh job payloads show runner_id: 0, empty runner name/group, and no execution steps, so these remain incomplete evidence.
The organization-required layer is also queued on the same exact head: required-workflow-bootstrap job 95234377394 in run 31975553200; scan-pr-queue job 95234376645 in run 31975553184; strix job 95234376591 in run 31975553207; noema-review job 95234376185 in run 31975553165; and close-empty job 95234375951 in run 31975553162. These exact-head checks are not predecessor evidence and are not merge-passing while queued.
A separate Analyze (python) check 95234374630 in run 31975552222 is also queued on 4561282..., but its returned pull_requests list is empty. Keep it as control-plane evidence until its owning workflow establishes authority rather than inferring authority from its display name.
All currently returned inline review threads on docs: record aFIPC required workflow coverage gap #124 are resolved; formal reviews are COMMENTED automation only, not a counted approval.

Additional exact-head evidence from the current OriginWeave MV3 integration root:

PR Refresh governance audit after clearfolio rollout #43 remains unchanged at exact head a4b1da1533aebd58956a265b6fd40bf5c7d5b3bb on protected main 0841d2ab3d8b5e60a03c0a8e818cf438e2716829; local CI, Manifest V3 Compatibility, SAST, and Security Scan are all successful.
Previously waiting strix job 95229578893 and coverage-source-tree job 95232903176 both eventually completed success without product-source churn.
A newly dispatched exact-head required opencode-review job 95238403791 in run 31973555801 is currently queued with runner_id: 0, empty runner name/group, and no steps. It is therefore incomplete current-head evidence, not a source finding and not a passing review gate.
This transition from earlier queued central jobs to success while a later exact-head central job again has no runner assignment further supports treating the recurring class as owner-side runner/capacity/control-plane state rather than leaf code failure. The clean Refresh governance audit after clearfolio rollout #43 source head must not be perturbed merely to retrigger it.

Fresh evidence after non-destructive protected-main reconciliation of two independent OriginWeave roots:

PR Ignore transient governance helper cancellations #46 (feat/sensitive-handle-use-reservation) is now exact head 10aaa324e55de8eac8b5e1fa1a88f6b55812603c, ahead 7 / behind 0 from protected main with merge base exactly 0841d2ab.... Native CI run 31978398999 is success: Rust contracts job 95241291802 completed repository contracts, formatting, locked workspace checks, full tests, strict Clippy and rustdoc; production-coverage job 95241291793 completed exact owned production function/line/region/branch enforcement. On the same exact clean head, SAST run 31978399007, Semgrep job 95241260989, remains queued with runner_id: 0, empty runner name/group, and no steps. Security Scan run 31978399004 is likewise queued; its returned Scorecard check 95241261482 is queued. The leaf branch was not changed to retrigger either scanner.
PR fix: keep OpenCode approval dispatch nonblocking #157 (fix/evidence-error-contract) is now exact head eac07a7125e541c4912b195a7522c319cb2b8792, ahead 5 / behind 0 from the same protected main with exact merge base 0841d2ab.... SAST run 31978497547, Semgrep job 95241499222, is queued with runner_id: 0, empty runner name/group, and no steps. Security Scan run 31978497543 is queued. Native CI run 31978497609 has begun executing, demonstrating that ordinary GitHub-hosted runner assignment can proceed for this same commit while the scanner lane remains unassigned.
The independent Ignore transient governance helper cancellations #46 and fix: keep OpenCode approval dispatch nonblocking #157 current-head observations reproduce the same control-plane symptom after each branch was cleanly reconciled to current protected main. They therefore strengthen the causal classification: runner_id: 0 scanner/required-workflow queue starvation is not evidence of a source defect and must not be converted into leaf code churn, catch-all exception suppression, or a passing result.

Owner-side acceptance remains unchanged but now binds to the current exact identities above: diagnose organization runner assignment/capacity/concurrency; preserve the sole current-head required evidence lanes; identify/cancel only provably obsolete work; expose queue age plus runner assignment in the machine-readable health snapshot; distinguish jobs with missing PR linkage instead of guessing from names; and let clean leaf heads wait/refetch/retry without no-op source churn. GREEN owner proof is real runner assignment and execution for the current-head jobs, or a health snapshot that surfaces an explicit external/manual capacity, billing, policy, or provider prerequisite without weakening merge gates.

Fresh Context Fabric reproduction adds two newly created repositories to the existing queue-starvation incident, with exact current-head and runner-assignment evidence rather than a leaf-source diagnosis.

ContextualWisdomLab/context-graph-contracts#4 exact head f34bd02eb3f78e1099a673a12d51892ff4487a9e, live base main@99cb5468ba3c15c5e79688f53dee74724fae2d13: repository CI run 31965340327 has been queued since 2026-08-16T18:41:37Z. All five jobs (95209547753, 95209547772, 95209547813, 95209547823, 95209547877) are queued, have runner_id=0, empty runner_name, and no steps. Draft PR chore: Record lack of UI codebase in palette journal #5 exact head e01896f639d31bcecd6d3224c625827d8d7ebf12 independently has CI run 31960192308 with all five jobs still queued since its earlier trigger, so this is cross-head within the same repository.
ContextualWisdomLab/enterprise-architecture-core#4 exact head 774c2a4ce12eb13a8596720472ac660d0ef580a1, live base main@1c0fa8b15ceb9e72186274aeb255d6777eb84ef4: CI run 31965349397 has seven queued jobs (95209569578, 95209569618, 95209569632, 95209569644, 95209569647, 95209569653, 95209569655), each with runner_id=0, empty runner name and no steps. Independent runtime-readiness run 31965349396, job 95209569108, has the same shape.

First failing boundary: GitHub-hosted ubuntu-latest runner assignment has not occurred; started_at equals creation time but is not assignment evidence when runner_id=0/runner name empty and steps=[]. No Context Fabric source execution has begun, so these queued states cannot falsifiably be attributed to Python/PostgreSQL/test failures. This is consistent with the OriginWeave/noema evidence already attached to #712 and strengthens the cross-repository control-plane hypothesis.

Falsifiable RCA split for the owner lane: distinguish hosted-runner capacity/assignment, organization billing or usage limits, runner-group/policy access, global/concurrency saturation, and stale run-state. Do not retrigger unchanged heads merely to manufacture activity. Smallest safe remedy remains the #712 read-only queue-health/control-plane diagnosis plus only the operator/control-plane action supported by its evidence; cancellation should be limited to proven obsolete runs, never these current-head lanes.

Required GREEN proof for Context Fabric after the control-plane remedy: the unchanged exact heads above must be assigned real runners (runner_id>0/non-empty runner identity), execute their actual steps, and reach terminal conclusions; then Context Fabric will freshly validate every repository and central required workflow on those same exact heads. Queued/pending/cancelled/skipped/absent evidence remains non-passing.

Fresh downstream evidence from ContextualWisdomLab/noema#400 narrows an additional queue-starvation/gate-composition case that the central owner should include in this incident without weakening any hard security gate.

Exact Noema state at handoff:

protected Noema main: eaeff88035e4c57c545d30b10970c5476fa2d765;
PR fix: tolerate manifestless OSV scans #400 exact head: 5b60f270463e4058ae44a4915972c7a0319a0398, base main, mergeable, non-Draft;
repository CI run 31962491331: terminal success;
reviewer-ci run 31962491488: terminal success;
required central Security Scan run 31962491343: still queued/non-terminal;
within that exact Security Scan run, trivy-fs job 95202490658, osv-scan 95202490670, and dependency-review 95202490710 are terminal success;
the remaining scorecard job 95202490669 is queued, has runner_id=0, empty runner_name, no steps, and has not produced Scorecard evidence.

The current central workflow intentionally configures Scorecard as soft/advisory at job semantics, while the Noema ruleset requires the whole central security-scan.yml@main workflow. Therefore a non-terminal advisory Scorecard job can hold the required workflow non-terminal even after the hard Trivy/OSV/dependency-review surfaces complete. This comment does not infer why GitHub has not assigned a runner; hosted-runner capacity/billing/policy/platform cause remains unresolved.

Please include this as a bounded central RCA/repair target in #712: determine whether advisory posture work should remain inside the required workflow's terminal dependency surface. A valid repair must preserve every hard Security Scan gate and fail-closed semantics; it must not translate missing Scorecard evidence into a clean Scorecard result. Candidate acceptance is a deterministic contract proving that (a) hard Trivy/OSV/dependency-review/other required security failures still fail the required workflow, (b) advisory Scorecard unavailability/queueing cannot indefinitely prevent the required security decision from reaching its governed terminal state, and (c) Scorecard evidence remains separately observable and non-fabricated. Revalidate with this unchanged Noema #400 head after any central integration; predecessor or synthetic status is not sufficient.

Fresh current-head queue-starvation evidence from the dedicated Context Fabric owner loop; no predecessor/status-only evidence is being substituted and no rerun/merge was requested.

ContextualWisdomLab/context-graph-contracts#4 exact head 25ffc97b2e25f2a7f9010fd7410104e655027696: workflow run 31970461357 (created 2026-08-16T20:24:43Z) remains queued. Current-head jobs observed queued include test (3.11) check 95222030105, test (3.13) 95222030219, test (3.14) 95222030123, and package 95222030094 (plus the same-run 3.12 matrix job). A fresh repository query for actions/runs?status=in_progress returned zero in-progress runs while these current-head jobs remain queued.
ContextualWisdomLab/enterprise-architecture-core#4 exact head 1c594e7480f80b5c036a103e58533fddd701327c: CI run 31970916787 and installed-process run 31970916784 (created 2026-08-16T20:34:05Z) remain queued. Current-head lanes include package check 95223133823, compose-runtime 95223133788, validate (3.11) 95223133794, the remaining Python matrix/migration jobs, and installed-process-readiness 95223133936.
ContextualWisdomLab/context-graph-contracts#5 exact head b902b38c24bfeb85d6ab8bbdf5e32ee490ba7fe6: run 31971194168 (created 2026-08-16T20:39:44Z) has all five current-head jobs queued; observed checks include test (3.11) 95223806917, test (3.12) 95223806935, test (3.13) 95223806890, and package 95223807047.

These heads contain newly repaired product defects/regressions and require their own fresh evidence before any Draft/merge state can advance. Please classify the first external queue boundary for these exact runs per #712 acceptance (runner/concurrency/billing/organization policy/other), preserve the sole current-head evidence, and cancel only genuinely obsolete/superseded work if found. Context Fabric will continue independent local work and will revalidate these exact heads once execution starts.

Fresh exact reproduction from OriginWeave PR #175 after a real source rebase/repair, 2026-08-17 KST:

protected main: 0841d2ab3d8b5e60a03c0a8e818cf438e2716829;
repaired current PR head: db724d4ec2ecda58f48463546e135d29ba03bd81, ahead of that live main with merge base equal to live main;
leaf CI 31987130156 completed success and Manifest V3 31987130192 completed success on this exact head;
SAST run 31987130159, job 95264011970, remains queued with runner_id: 0, empty runner name, zero steps;
Security Scan run 31987130160 is likewise still queued on the same exact clean head.

This rules out a compile/test failure as the first boundary for the queued SAST lane and gives the queue-health implementation another current-head runner_id=0 specimen. Please include this exact run/job identity in the owner-side queue classifier/regression corpus and preserve fail-closed semantics: no leaf retrigger commit, no queued→success coercion. Acceptance for this specimen is either runner assignment/execution on the unchanged head or an owner-side classified external/capacity state with bounded retry/queue-age evidence.

Additional current-head reproduction from OriginWeave after a real test-first product repair, not a retrigger-only commit:

OriginWeave#175 exact head is now 6b7cfd74486932541bfedc66c5751a98804a0678 on protected main 0841d2ab3d8b5e60a03c0a8e818cf438e2716829.
The preceding test-only head 4b36b53f6be66bafd5c43e7bc62b2cbd97460ccb produced the intended RED in CI run 31991495632, Rust-contracts job 95275765872, at Run tests; production then changed only the reviewed Chrome compatibility classification and exact-head CI 31991567854 is GREEN. Manifest V3 Compatibility 31991567888 is also GREEN.
On the same exact clean production head, SAST run 31991567928 remains queued: Semgrep job 95275958416 has runner_id: 0, empty runner name/group, and no steps.
Security Scan run 31991567894 likewise remains queued: dependency-review 95275958441, OSV 95275958451, Trivy FS 95275958452, and Scorecard 95275958604 each have runner_id: 0, empty runner name/group, and no steps.

This fresh head again separates executable leaf evidence from the unassigned scanner lane: normal CI and pinned Chromium execute successfully while the scanner jobs never receive a runner. Keep these queued jobs non-passing, do not churn the leaf source to retrigger them, and include these exact identities in the owner-side queue-health/runner-assignment diagnosis.

Agent: Codex
Started: 2026-08-19T09:48:00Z
Scope: read-only current-head Actions queue health snapshot, deterministic JSON/accessible HTML report, no cancellation or merge-gate weakening in this slice.

Agent: Codex
PR: #1142
Current branch: agent/actions-queue-health
Current HEAD: 77431bacc2cf15e49d56a6854d117c8d67935a5d

Delivered the read-only first slice: explicit allowlist for .github, TEPP, naruon, and contextual-orchestrator; bounded paginated Actions/PR/job evidence; exact open-PR head classification; queued/in-progress runner evidence; deterministic JSON and accessible HTML artifacts; 900-second SLO warning; no cancellation, dispatch, branch, or merge mutation.

Local evidence: 1254 pytest tests plus 16 subtests passed; focused queue-health tests passed; repository branch/statement coverage 100%; interrogate 100%; changed-file Ruff/compileall/actionlint checks passed. Live per-repository read probes completed for all four allowlisted repositories; an aggregate read can fail closed if pagination observes a malformed/transient non-atomic PR page.

PR remains subject to current-head required Checks and independent approval. No merge or queue cancellation was performed.

Live read-only diagnostic at 2026-08-19T10:26:54Z (GitHub Actions REST total_count, per_page=1):

Repository	queued	in_progress
`ContextualWisdomLab/.github`	813	38
`ContextualWisdomLab/TEPP`	6	1
`ContextualWisdomLab/contextual-orchestrator`	6	0
`ContextualWisdomLab/naruon`	36	10
Total	861	49

This confirms the queue-starvation external condition independently of source tests. PR #1142 required runs are still queued at current HEAD 77431bacc2cf15e49d56a6854d117c8d67935a5d; no run was cancelled and no merge gate was weakened.

LineageWeave current-head canary (2026-09-23): #1126 exact 1515b872d32b7e943c6c09b0f7da2806fe1224ac remains Draft. Tests 35876759420 completed skipped by Draft policy; SAST 35876759314 is queued and job 107234564791 has runner_id=null, empty runner/group, and steps=[] despite started_at=2026-09-23T14:47:22Z. Security 35876759503 is queued and CodeQL PR 35876759391 pending on the same exact head. This is fresh current-head queue evidence, not a request to Ready-cycle, wake-commit, blind-rerun, or synthesize status. Please keep it in the canonical queue-starvation diagnosis/health snapshot.

OriginWeave #37 exact CodeQL owner-path canary has completed runner admission and now isolates a deterministic central control-plane defect. Target remains ContextualWisdomLab/OriginWeave#37@219b43bfa87ab4fd90a77bb962df2797925bf661 over protected base 87c4daa1830bac5a5228b6036752ad5633232085; consumer CodeQL run is 35816715522 and canonical producer run is .github 35870614075.

The producer is no longer a queue-starvation specimen. validate-dispatch 107213403257 completed SUCCESS. All three language scans obtained hosted runners and reached the actual analysis path: Actions 107322434021, JavaScript/TypeScript 107322434023, and Python 107322434093 each completed Initialize CodeQL, Perform CodeQL Analysis, and Enforce CodeQL Medium+ SARIF gate successfully. All three then failed at the same step, Verify GHAS base/head CodeQL configuration identity. The terminal settle exact required run job 107392374879 also obtained a runner and failed specifically at Settle exact CodeQL required run after token exchange succeeded.

Therefore this exact specimen should be removed from the unresolved runner-admission set. It now belongs to the CodeQL dispatch/terminal-publication owner path: scan execution and SARIF policy are GREEN, while GHAS base/head configuration identity and exact-run settlement are RED. No OriginWeave source churn, consumer workflow copy, blind rerun, wake commit, timeout inflation, or required-check bypass is warranted from this evidence.

DiskSage #461 runner-admission specimen refreshed to current exact c956a5d0a402e852903b7f0ea43c4e4dac80221e; this supersedes the older 121ac434... snapshot and its then-valid source prediction.

Test run 35908938050 is still queued. Windows windows-home-resolution job 107343575265 completed SUCCESS on hosted runner 1002111508, while Ubuntu llm-engine-build 107343575106 and Ubuntu main test 107343575273 remain pre-execution queued with runner_id=0, empty runner identity, and steps=[]. SAST Semgrep 35908938107 is pending; CodeQL PR 35908938216 and Security Scan 35908938217 are queued. Release 35908938054 is skipped as expected for a PR exact.

The old 121ac... prediction that CargoTargetReclaimError was absent is no longer current. By c956..., DiskSage has a canonical typed CargoTargetReclaimError::PartialCleanup / versioned partial-clean receipt, maps typed CleanupFailure::Partial into the owner error without helper→owner String erasure, passes that typed error to the CLI, and serializes the receipt once at the presentation boundary. The latest descendants also preserve irreversible partial-clean evidence if best-effort residual measurement fails and derive optional target-view reduction from canonical before/after measurements while keeping ledger_reclaim_bytes = 0.

Therefore c956... currently has neither hosted Unix RED nor hosted Unix GREEN. Keep it as runner-admission evidence only until an Ubuntu job actually executes. If execution later produces a compile/test failure, reclassify that job as leaf source evidence and retire the runner-starvation classification for that lane. No blind rerun, no-op wake commit, runner-selector change, timeout inflation, or gate weakening is warranted.

Fresh Wardnet current-head queue specimen for canonical runner owner diagnosis; no leaf rerun/churn requested.

repo/PR: ContextualWisdomLab/wardnet#443
exact head: 239c747848a8241cbbccd66e6f91d20725860c51
source character: test-only successor; production remains byte-identical to protected Wardnet main@f8260f1e03836039ff9463dd99fa982e4e270c4b
CI run: 35880184584
rust job: 107246315069
observed state: queued, no conclusion, returned steps: null before checkout/execution
same-head companion runs currently queued: Security 35880184517, SAST 35880184578, CodeQL PR 35880184686

This successor adds two real loopback hostile contracts (slow-consumer bounded backpressure and four concurrent held streams) on top of predecessor semantic RED; it is intentionally not treated as RED/GREEN until the current head actually executes. Owner GREEN acceptance remains real runner assignment/execution for the current exact head, or an explicit owner-side capacity/billing/policy/provider prerequisite surfaced without weakening required gates. Do not solve this by no-op leaf commits, blind reruns, copied central workflows, synthetic status or bypass.

Fresh pingora-gateway runner-admission specimen. Current #22 exact b082fe2948104b3afd6c454d9c187877a0659671 has CI 35881360004 and Supply 35881360015; CI jobs 107250353453 test, 107250353882 load-contract, 107250354209 oci-runtime and Supply 107250352067 candidate-evidence are all queued, steps=[], runner_id=null, no conclusion. This is current-head incomplete evidence, not GREEN and not execution RED. Intermediate Repair 96 commits were superseded while the lane moved, and the first test-first commit 65884a95... also had a local Rust lifetime defect, so do not use it as a runner-failure specimen. By contrast foundation exact 0cf3b437... has actually executed Security FAILURE + CodeQL FAILURE + SAST SUCCESS. Please keep these classes distinct in queue-health RCA and preserve the fail-closed rule: unavailable runner != success.

Fresh exact-head canary from LineageWeave #1125 after a substantive PostgreSQL migration repair: head 2f2bac4d999482a68422811b20c92dc715b0375d, SAST run 35888625033, job 107275130150. The job is still queued with runner_id=0, empty runner/group names and steps=[] (ubuntu-24.04 label). Tests on the same head are Draft-policy skipped; Security 35888625098 is queued and CodeQL 35888625166 pending. No leaf rerun, wake commit, Ready/Draft cycle, runner-selector change or synthetic required status was used.

Fresh pingora-gateway queue specimen. Test-first #22 head d6a3cc95f28866b320568953ebb6ddf69c8f5c6d added the inherited Cargo/rustc compiler-environment sanitizer contract but was superseded before runner execution: CI 35888236825 test 107273816575, load-contract 107273816879, oci-runtime 107273817098 all ended cancelled with steps=[], runner_id=0; do not classify that as execution RED. Current causal head is e23d2edabfddb0986545e37aa133c0130a169c61: CI 35888554455 test 107274925825, load-contract 107274926276, oci-runtime 107274926318 and Supply 35888554433 candidate-evidence 107274897938 are queued with steps=[] and no runner assignment. That is current-head incomplete evidence, not GREEN. Keep it distinct from foundation #1, whose Security/CodeQL lanes actually executed and failed.

Superseding pingora-gateway runner specimen. Current #22 exact is a2f347b575436516412484b6fd272f28ba7cd1b7; a2f347b5... is comment-only reconciliation after Repair 97 and does not change the executable contract. Current CI 35889792696: test 107279083000, load-contract 107279083246, oci-runtime 107279083533; Supply 35889792980: candidate-evidence 107279073971. All are queued with steps=[], no runner assignment, no terminal conclusion. Treat as incomplete current-head evidence, not GREEN. Earlier test-first d6a3cc95... and causal e23d2eda... runs were superseded/cancelled pre-runner; keep those distinct from foundation #1 Security/CodeQL lanes that actually executed and failed.

LineageWeave current-head canary (2026-09-24 KST): PR #1126 exact 1b4af6780a8b81dbf0391fa422cbdf0761fd6325 remains Draft by owner contract. SAST run 35892427881, job 107287947202 is queued on ubuntu-24.04 with runner_id=null, no runner/group and steps=[]; CodeQL run 35892427728 and Security run 35892427669 are also queued. Tests 35892427701 is intentionally skipped by Draft policy. No Ready/Draft cycling, wake commit, blind rerun, runner-selector change, synthetic status, or gate weakening was used. Treat this as current-head required evidence unavailable until the canonical queue owner recovers runner admission.

Superseding the immediately preceding LineageWeave#1126 canary: current exact head is now 3e38a2847c13ca1424e93c328a8fb139993930b7 (new owner RED; still Draft). SAST run 35893108657, job 107290202134 is again pre-runner queued on ubuntu-24.04 with runner_id=0, empty runner/group and steps=[]; Security 35893108549 and CodeQL 35893108634 are queued, while Tests 35893108581 is Draft-policy skipped. No rerun/wake/Ready-cycle/selector change. The earlier 1b4af678... specimen is obsolete for exact-head acceptance; this one is the live queue specimen, though the source head itself is intentionally RED until its App.tsx causal repair lands.

Fresh Wardnet queue specimen after a substantive owner repair, not a no-op wake. PR #441 exact head 29a35163fc5bab860f247a68a6b1d5f797d73182 (fix(gateway): reject ambiguous singleton responses) has CI 35893229211, rust job 107290605213, still queued with steps=[], runner_id=0, empty runner/group identity on ubuntu-24.04; Fuzz 35893229215 is queued on the same exact head. Its test-only children were non-force restacked to that repair and are likewise current-head incomplete evidence: #446 exact 8ae44f90dec252140ad65075a96ad1c8b256f0ea, CI 35893328787; #448 exact 44c03a0fe83b7199ebad43cff0e009646f7554f1, CI 35893375073. Older #446 exact predecessor did obtain a runner and produced valid semantic RED, proving the repository test path itself is executable when capacity is assigned. Keep all current runs fail-closed/incomplete; do not blind-rerun, synthesize status, or churn Wardnet source. RED→GREEN acceptance for this central owner remains: current exact heads acquire a runner, execute deterministic/security checks to terminal result, with obsolete predecessor runs cancelled only when safely superseded.

Fresh pingora-gateway runner-admission specimen. Repair 98 test-first 4a6fa8d0fd651fdecf630ce6bc202b8801d617ec produced CI 35895189238; after the successor moved, load-contract 107297255268, oci-runtime 107297255547, and test 107297255744 all completed cancelled with steps=[] and runner_id=0, so that is pre-runner cancellation, not execution RED. Current causal exact 821879b1e095e7271fbbea0b102f0ea038b26373 has CI 35895396320 and Supply 35895396342; test 107297946249, load-contract 107297946228, oci-runtime 107297946283, and candidate-evidence 107297945489 are queued with steps=[], runner_id=0, no terminal conclusion, so this is incomplete runner-admission evidence, not GREEN. Keep that distinct from foundation #1, where Security and CodeQL actually completed FAILURE, SAST completed SUCCESS, and CI/Supply were intentionally SKIPPED by Draft lifecycle.

Fresh ConceptWeave Product-ruleset queue specimen on exact current owner head.

Target PR: ContextualWisdomLab/.github#2350
Target head: a1628e75dc056098f4ea2d03ec799f2026025e84
Owner workflow: ConceptWeave Product Ruleset Reconcile run 35891410528
Job: validate / 107284521412
Attempt: 1

Fresh 2026-09-24 KST read:

status queued, conclusion null;
runner_id=0, empty runner/group names;
steps=[];
label ubuntu-24.04;
created/started timestamp 2026-09-23T16:49:33Z.

This is the sole current-head owner evidence for the mutation-boundary repair already staged on #2350. The preceding contract head fdbe09a... produced the intended terminal lifecycle RED; a1628e7... is the minimal ordinary-forward production repair and is intentionally being held unchanged until this exact run settles. Review 5294002721 records the next, separate TOCTOU contract and must not be implemented by overwriting this queued evidence.

The same queue class is also visible one stage downstream of ConceptWeave Product #35: .github CodeQL Scan Dispatch run 35870670165, validate-dispatch job 107213600319, is still queued with runner_id=0 and steps=[] for exact ConceptWeave#35@d7b7e30... / base f4f440dd... / required 35825042996.

GitHub's public status currently reports Actions operational, so there is no public incident evidence to substitute for the organization-specific admission diagnosis. Do not infer a leaf source defect from either queued job. No no-op wake, blind/manual rerun, runner-selector change, timeout inflation, synthetic status, or gate weakening is warranted while these exact current-head jobs remain live.

LineageWeave#1126 current-head canary after source repair: db542f3da361321462d540de550f5b84e170a93d. Draft-policy Tests run 35899925429 is skipped; SAST 35899925758 remains queued and job 107313179818 has not started (steps=null), Security 35899925551 queued, CodeQL 35899925673 pending. No Ready cycling, wake commit, blind rerun, runner-selector change, or synthetic status was used. This is another exact-head specimen for the existing queue-starvation owner issue, not a request to weaken LineageWeave gates.

Fresh Wardnet sole-ledger current-head queue specimen after substantive status/documentation repair (not a wake/no-op commit): wardnet#130@2ea72441c56cd30f833b1bf9d64ae7ebf3d58951 changes only docs/product-technical-gap-baseline.md to reconcile live #441/#446/#448 heads and evidence. Exact CI 35900514711, rust job 107315202314, is queued on ubuntu-24.04 with steps=[] and no assigned runner identity; exact Security Scan 35900514729, SAST 35900514689, and CodeQL PR 35900514695 are also queued. Base remains protected main@f8260f1e03836039ff9463dd99fa982e4e270c4b; PR is Draft/mergeable. Keep all lanes fail-closed/incomplete; do not blind-rerun, synthesize status, alter Wardnet source/workflow selectors, or cancel sole current-head evidence. Owner GREEN remains real runner assignment/execution to terminal results, with obsolete predecessor work cancelled only when safely superseded.

Navigation Menu

Repository navigation

ops: diagnose and bound organization GitHub Actions queue starvation #712

Operational gap

Required diagnosis

Acceptance

Related work

2501 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

ops: diagnose and bound organization GitHub Actions queue starvation #712

Description

Operational gap

Required diagnosis

Acceptance

Related work

Activity

seonghobae commented on Aug 3, 2026

seonghobae commented on Aug 3, 2026

seonghobae commented on Aug 15, 2026

seonghobae commented on Aug 16, 2026

seonghobae commented on Aug 16, 2026

seonghobae commented on Aug 16, 2026

seonghobae commented on Aug 16, 2026

seonghobae commented on Aug 17, 2026

seonghobae commented on Aug 17, 2026

seonghobae commented on Aug 19, 2026

seonghobae commented on Aug 19, 2026

seonghobae commented on Aug 19, 2026

2501 remaining items

seonghobae commented on Sep 23, 2026

seonghobae commented on Sep 23, 2026

seonghobae commented on Sep 23, 2026

seonghobae commented on Sep 23, 2026

seonghobae commented on Sep 23, 2026

seonghobae commented on Sep 23, 2026

seonghobae commented on Sep 23, 2026

seonghobae commented on Sep 23, 2026

seonghobae commented on Sep 23, 2026

seonghobae commented on Sep 23, 2026

seonghobae commented on Sep 23, 2026

seonghobae commented on Sep 23, 2026

seonghobae commented on Sep 23, 2026

seonghobae commented on Sep 23, 2026

seonghobae commented on Sep 23, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions