nddev-linux-release does not refill; queued Android job starved after Rust

[rldyourmnd]

Summary

The published nddev-linux-release class served one short job and then left a longer Android job unassigned. NDDev-it-com/almaty-libraries currently lists zero registered runners.

This is not the 2026-09-03 lost-communication failure (server-nddev-amsterdam#149): the Android job never received a runner.

Observed 2026-09-06 (UTC)

Repository: NDDev-it-com/almaty-libraries
PR: https://github.com/NDDev-it-com/almaty-libraries/pull/1217
quality-gates run: https://github.com/NDDev-it-com/almaty-libraries/actions/runs/34005712826
SHA: 3045ad5aa

Time	Event
02:13:30Z	Mobile Android build (bounded local debug APK) entered queued on nddev-linux-release. runner_name empty, no steps.
02:26:39Z	Rust (telemetry-engine) also nddev-linux-release was assigned nddev-ysc5yrnv6zmu — 13 minutes after Android was already waiting.
02:33:48Z	Rust completed success.
02:50:50Z	Android still queued. GET /repos/NDDev-it-com/almaty-libraries/actions/runners → total_count: 0.

What this is not

• Not a workflow timeout (Android job bound is 60 minutes; it has not started).
• Not Gradle/application failure (no steps ran).
• Not “label unpublished”: Rust on the same label was admitted and finished.

Ask

1. Why a later nddev-linux-release job was assigned while an earlier queued job on the same label was skipped.
2. Why the scale-set did not create a second instance (or refill after Rust) for the still-queued Android job.
3. Confirm whether nddev-linux-release is actually serving for jobs whose wall-clock is expected to exceed ~15 minutes (bounded local debug APK).

No secrets in this report.

[rldyourmnd]

Host/fleet observation: https://github.com/NDDev-it-com/server-nddev-amsterdam/issues/154
Related lost-comms (different symptom): https://github.com/NDDev-it-com/server-nddev-amsterdam/issues/149

[rldyourmnd]

2026-09-06 second wave on the same PR 1217 run 34005712826:

Android did get nddev-linux-release runner nddev-ph8opaznkrjl at 02:51:13Z after ~38 minutes queued. Checkout, hermetic SDK and Flutter/codegen succeeded. Gradle step Build bounded local debug APK started 02:52:46Z with conclusion=null / completed_at=null. Job marked completed failure 03:09:14Z (~18m wall, ~16m into Gradle). Job logs 404 (BlobNotFound) — the worker vanished before uploading logs.

This is the same lost-communication-during-Gradle pattern as this issue, not an application BUILD FAILED. Candidate certified then failed because the mobile gate requires Android success.

Job: https://github.com/NDDev-it-com/almaty-libraries/actions/runs/34005712826/job/101412960130

[rldyourmnd]

2026-09-06 third wave: Android does execute on nddev-linux-release once a runner is assigned. The skip/refill bug is still live.

Execution works (8192 envelope)

Three bounded local debug APK successes after the 6144 lost-comms:

When (UTC)	Job	Runner	Gradle	Result
03:13:46–03:22:45	101419702872 (PR 1217 rerun)	nddev-uuoujn468zjl	assembleLocalDebug 436.6s	success
05:10:16–05:19:36	34009427188 (main workflow_dispatch)	nddev-qgwacklzzbp9	~9 min	success
05:24:10–05:33:07	101421189075 (PR 1218)	nddev-vgnmfcrnbe3e	425.6s, ✓ Built …/app-local-debug.apk	success

Ask #3 from the original report is answered: the class can serve a ~9 minute Gradle job.

Assignment still skips the longest-waiting Android job

PR 1218 Android entered queued at 03:25:02Z and was not assigned until 05:24:10Z (119 minutes). During that wait, later jobs on the same label were served first:

Time	Event
03:25:02Z	PR 1218 Android queued (nddev-linux-release, no runner).
03:36:44Z	main workflow_dispatch quality-gates created (later than 1218).
03:46:58–03:52:25Z	Dependabot Rust assigned nddev-s9xdrpxog1vc and succeeded.
04:08:33–04:15:46Z	Dependabot Rust assigned nddev-ecn5zcimspnw and succeeded.
04:39:00Z	Four Dependabot Android jobs cancelled with empty runner_name (never assigned).
04:39:49–04:45:28Z	main Rust assigned nddev-rjw89imvt4ky and succeeded.
05:03:45–05:09:46Z	PR 1218 Rust assigned nddev-isjzwpaca9fy and succeeded.
05:10:16Z	Later main Android assigned nddev-qgwacklzzbp9 — 14 minutes before the earlier 1218 Android.
05:19:36Z	main Android success.
05:24:10Z	PR 1218 Android finally assigned nddev-vgnmfcrnbe3e.

GET /repos/NDDev-it-com/almaty-libraries/actions/runners still returns total_count: 0 while an org ephemeral runner is executing (expected if registration is org-level; still the only public signal we have).

Host observation: https://github.com/NDDev-it-com/server-nddev-amsterdam/issues/154
Lost-comms (different symptom, still open): https://github.com/NDDev-it-com/server-nddev-amsterdam/issues/149

No secrets in this report.

[rldyourmnd]

Related, different symptom than skip/refill: PR 1219 Android on nddev-gkvklzcsiv18 did get a runner at 06:32:06Z, then the worker received a shutdown signal at 06:48:52Z (~15 min into Gradle). Job logs are present (The runner has received a shutdown signal / The operation was canceled). Tracking that on https://github.com/NDDev-it-com/server-nddev-amsterdam/issues/149

This wave is not additional skip/refill evidence. 1218 still waited 119 minutes earlier today.

No secrets.

[rldyourmnd]

Product: #407
Host: https://github.com/NDDev-it-com/server-nddev-amsterdam/issues/155

Same skip/refill as this issue, different scale-set (nddev-priority-integration, pytest/image jobs, not Android). Do not merge/close as duplicate: #406 is nddev-linux-release. Live starve still open at 09:45Z (pytest shard 4/4 queued 68 min, then 0 workers).

[rldyourmnd]

Almaty-side mitigation (does not fix GARM refill): quality-gates will keep nddev-linux-release exclusive to the bounded Android APK. Rust moves off that label onto nddev-priority-integration. Infra-only PRs skip the APK.

PR: https://github.com/NDDev-it-com/almaty-libraries/pull/NEW

Refill/FIFO on the scale-set is still this issue / #407.

[rldyourmnd]

Correction: Almaty PR is https://github.com/NDDev-it-com/almaty-libraries/pull/1224 (not NEW). Still does not fix GARM refill.