What happens
Builds on releases/25.3.14 hang in Check out repository code. The Altinity/checkout action clones ~124 submodules in parallel (xargs --max-procs=100). One of them stops receiving data and nothing gives up — git has no timeout by default, so the job sits there until it is killed.
The stuck submodule is always one of the big ones: contrib/aws, boost, llvm-project, icu, rust_vendor, google-cloud-cpp.
| Run |
When (UTC) |
Result |
| 35063459169 |
2026-09-16 06:29 |
7 builds cancelled at 6 h |
| 35160470149 |
2026-09-16 23:05 |
7 builds cancelled at 6 h, one runner lost contact |
| 35258278324 |
2026-09-17 18:25 |
3 builds cancelled at 2.5 h |
It hits static and ephemeral runners alike. Signature in the log: the last line is a Submodule path '...': checked out ..., then silence, then ##[error]The operation was canceled. The runner then kills an orphan git-remote-https.
Each hang blocks the whole run — in 35160470149 the ~60 regression jobs waited six hours to fail in 20 seconds.
What #2402 fixed, and what it did not
#2402 added timeout-minutes: 150. That works: the third run died at 2 h 30 instead of 6 h.
It also set GIT_HTTP_LOW_SPEED_LIMIT=1000 and GIT_HTTP_LOW_SPEED_TIME=60, meant to make git abort the stalled fetch so .github/retry.sh would retry it. That part does not work. In Build-package_release both variables are in the step env, but the log has no curl 28, no error of any kind in 2 h 31, and retry.sh invoked the stuck submodule exactly once.
The setting itself is fine — it aborts a real clone locally:
error: RPC failed; curl 28 Operation too slow. Less than 100000000 bytes/sec transferred the last 2 seconds
Note RPC: curl only measures speed while a transfer is running. It does not measure during connect, during the TLS handshake, or while git blocks anywhere else. Our stall sits outside that window, so the timer never starts.
Possible fixes
- Cap the parallelism. 100 concurrent clones from one IP is the likely trigger.
Altinity/checkout's main already moves to $nproc and retry.sh 5 60, but that commit's YAML does not parse — one line is indented 7 spaces instead of 8 — and no branch pins it. Port the change onto the pinned 2023 SHA with an explicit cap, then bump the pin in reusable_build.yml only; it is the only workflow that checks out submodules.
- Wrap the command in
timeout. It does not care what phase git is in, and .github/retry.sh lives in this repo, so it can opt in through an env var. Untested: killing git mid-clone leaves a partial .git/modules/<path>, so the retry may then fail for a different reason.
Not seen since 2026-09-17.
What happens
Builds on
releases/25.3.14hang inCheck out repository code. TheAltinity/checkoutaction clones ~124 submodules in parallel (xargs --max-procs=100). One of them stops receiving data and nothing gives up — git has no timeout by default, so the job sits there until it is killed.The stuck submodule is always one of the big ones:
contrib/aws,boost,llvm-project,icu,rust_vendor,google-cloud-cpp.It hits static and ephemeral runners alike. Signature in the log: the last line is a
Submodule path '...': checked out ..., then silence, then##[error]The operation was canceled.The runner then kills an orphangit-remote-https.Each hang blocks the whole run — in 35160470149 the ~60 regression jobs waited six hours to fail in 20 seconds.
What #2402 fixed, and what it did not
#2402 added
timeout-minutes: 150. That works: the third run died at 2 h 30 instead of 6 h.It also set
GIT_HTTP_LOW_SPEED_LIMIT=1000andGIT_HTTP_LOW_SPEED_TIME=60, meant to make git abort the stalled fetch so.github/retry.shwould retry it. That part does not work. In Build-package_release both variables are in the step env, but the log has nocurl 28, no error of any kind in 2 h 31, andretry.shinvoked the stuck submodule exactly once.The setting itself is fine — it aborts a real clone locally:
Note RPC: curl only measures speed while a transfer is running. It does not measure during connect, during the TLS handshake, or while git blocks anywhere else. Our stall sits outside that window, so the timer never starts.
Possible fixes
Altinity/checkout'smainalready moves to$nprocandretry.sh 5 60, but that commit's YAML does not parse — one line is indented 7 spaces instead of 8 — and no branch pins it. Port the change onto the pinned 2023 SHA with an explicit cap, then bump the pin inreusable_build.ymlonly; it is the only workflow that checks out submodules.timeout. It does not care what phase git is in, and.github/retry.shlives in this repo, so it can opt in through an env var. Untested: killing git mid-clone leaves a partial.git/modules/<path>, so the retry may then fail for a different reason.Not seen since 2026-09-17.