Skip to content

fix(server): server updates retry a release download that stalls - #37155

Open
Sidreyas wants to merge 1 commit into
pingdotgg:mainfrom
Sidreyas:sidreyas/fix-36877-retry-download-stalls
Open

Sidreyas wants to merge 1 commit into
pingdotgg:mainfrom
Sidreyas:sidreyas/fix-36877-retry-download-stalls

Conversation

@Sidreyas

Copy link
Copy Markdown
Contributor

Problem

A server update fails as soon as the release download stalls once. fetchReleaseAsset (cloud.pinned_runtime.fetch_release_asset) applies the 1-minute PINNED_RUNTIME_DOWNLOAD_STALL_TIMEOUT to the response body, but a stall raised the same Cause.TimeoutError as a host that never answers, and isTransientDownloadFailure returns false for every TimeoutError. So DOWNLOAD_RETRY never retried a stall. On a slow link to GitHub release assets, one stall anywhere in the archive download failed the update with Pinned runtime install failed while downloading the t3 release archive (timed out)., which is the log in #36877.

Change

apps/server/src/cloud/pinnedRuntime.ts: the body stall now fails with its own module-private ReleaseDownloadStalledError, and isTransientDownloadFailure treats it as transient. This matches how a body that fails after the host answered (DecodeError) is already treated.

ensurePinnedRuntimeInstalled is shared, so remote server updates, t3 update and the service install all get this.

Scope

Refs #36877. This fixes only the first point of that issue: a stall is now transient and retried. Two parts are left:

  • Range resume. Each retry still downloads from byte 0. Resuming needs a staging file or a kept buffer, a Range/If-Range request, handling of a 200 that ignores the range, and new tests. That is a second change to the download path, so it belongs in its own PR. With this fix, a link that stalls now and then finishes once one attempt gets through without a stall.
  • The desktop Electron updater (net::ERR_TIMED_OUT) is a separate code path, and this PR does not touch it.

Verification

End to end, real HTTP. The real ensurePinnedRuntimeInstalled → fetchReleaseAsset path ran in Node with:

  • the real FetchHttpClient layer that t3 update uses
  • the real ProcessRunner, which extracts with System32 tar.exe
  • a --version validate like the one in t3 update

The release source was a local http.createServer mirror passed as releaseBaseUrl. Its fixture is a real win32-x64 release zip (39.5 MB, node.exe packaged as t3.exe) with its SHA256SUMS. The first archive response sends the headers and 1 MiB, then stalls for 75 s. Later requests get the full body. The timeouts are the real ones, with no shortened seam. Windows 11, Node 26.

Before, origin/main e101cae: one stall fails the install after a single attempt.

[+   0.7s] mirror: GET /v26.12.1/t3-26.12.1-win32-x64.zip -> 200 (39488544 bytes), sends 1 MiB then stalls 75s
[+   0.7s] progress: download 65120/39488544 bytes
[+  60.8s] mirror: stalled archive connection closed
[+  60.8s] RESULT: FAILED
PinnedRuntimeInstallError: Pinned runtime install failed while downloading the t3 release archive (timed out).
[+  60.8s] archive requests served: 1
exit code: 1

After, this branch: the stall is retried 2 s later and the install completes.

[+   1.5s] mirror: GET /v26.12.1/t3-26.12.1-win32-x64.zip -> 200 (39488544 bytes), sends 1 MiB then stalls 75s
[+   1.5s] progress: download 65376/39488544 bytes
[+  61.7s] mirror: stalled archive connection closed
[+  63.9s] mirror: GET /v26.12.1/t3-26.12.1-win32-x64.zip -> 200 (39488544 bytes, full body)
[+  65.2s] progress: download 39488544/39488544 bytes
[+  65.3s] progress: verify
[+  65.9s] progress: extract
[+  68.3s] progress: validate
[+  72.1s] validate: t3.exe --version -> v26.12.1
[+  72.1s] RESULT: installed runtime\versions\26.12.1\t3.exe
[+  72.1s] archive requests served: 2
exit code: 0

Unit tests in apps/server/src/cloud/pinnedRuntime.test.ts:

  • New: an archive body that sends 4 bytes and then stalls is retried, and the install completes. It fails on main with the production error, PinnedRuntimeInstallError: Pinned runtime install failed while downloading the t3 release archive (timed out)., and passes here.
  • New: a host that never answers makes one request and fails with downloading the t3 release checksums (timed out). This pins the side of the rule that must not retry, and it passes on main too.

Commands, from apps/server:

  • pnpm exec vp test run src/cloud/pinnedRuntime.test.ts src/cloud/selfUpdate.test.ts

    • Branch: 36 passed, 11 failed.
    • main: 34 passed, 11 failed, the same 11 tests.
    • All 11 are prune tests that fail with symlink EPERM on this Windows machine. This change does not touch them.
  • pnpm exec vp run typecheck: passes.

  • vp run check:changed: format, lint, knip 1/2, graphql and typecheck pass.

  • The pre-commit hook could not run in this worktree: its shared core.hooksPath loses the drive letter under Git Bash. Running its vp staged directly, vp fmt and vp lint passed on the staged files, but knip:check ran out of memory locally (RangeError: Array buffer allocation failed in oxc-parser). After these manual checks, the commit was made with the hook skipped.

Not checked here:

  • knip: 2/2 in check:changed hit its 900 s limit, and the hook's knip:check ran out of memory. CI runs it. The change adds no exports.
  • The Linux boot-service path from the issue was not run. It is the same shared function.

Written by Claude Opus 5.5 in Claude Code via T3 Code.

🤖 Generated with Claude Code

A pinned runtime download whose response body stops arriving for the
1-minute stall timeout failed the whole update: the stall raised the
same TimeoutError as a host that never answers, and
isTransientDownloadFailure treats that as fatal. The stall now fails
with its own error, which the existing retry schedule (3 more attempts,
2/4/8 s apart) retries like a body that fails after the host answered.
No answer at all still fails at once, and the error after the last
attempt still reads "(timed out)".

Refs pingdotgg#36877

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 15, 2026 10:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@github-actions github-actions Bot added vouch:unvouched Author is not yet trusted in the VOUCHED list. size:S 10-29 changed lines (additions + deletions). labels Nov 15, 2026
@macroscopeapp

macroscopeapp Bot commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Approved at e90ca3a

Macroscope's review found this PR approvable — Stalled release downloads now retry within the existing bounded retry policy, while unanswered requests still fail without retries. Integrity checks and staged installation remain unchanged, limiting the impact to download recovery and waiting time.

You can add or adjust custom eligibility rules. Learn more.

@coderabbitai

coderabbitai Bot commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

Pinned runtime downloads now distinguish response-body stalls from request timeouts. Body stalls are retried under the transient-failure policy and reported with a timeout step label. Tests cover a retried body stall and a request timeout that is not retried.

Changes: Pinned runtime download timeout handling

Layer / File(s) Summary
Classify and test download timeouts
apps/server/src/cloud/pinnedRuntime.ts, apps/server/src/cloud/pinnedRuntime.test.ts
Response-body stalls use a distinct error that is retried. Request-level timeouts remain non-retriable. Install errors label both cases as timed out. Tests verify that a stalled response is retried and a request that never responds is requested only once.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: marve10s, shivamhwp, juliusmarminge

Merge Risk: Minimal · up to e90ca

Pinned runtime downloads that stall mid-body are now retried, while requests that receive no response still fail. No merge-blocking risk was found.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main change: a server update now retries a release download when the body stalls. It follows the conventional commit form with a clear scope. It is specific to the change. It is a …
Description check Passed The description covers the problem, the change, scope, and verification. It has the required sections from the template and gives detailed evidence, including end-to-end logs and unit test results. Th…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the main change: a server update now retries a release download when the body stalls. It follows the conventional commit form with a clear scope. It is specific to the change. It is a little wordy because "server" appears twice, but the meaning is clear.

Full details: Description check

Explanation

The description covers the problem, the change, scope, and verification. It has the required sections from the template and gives detailed evidence, including end-to-end logs and unit test results. The Scope section links issue #36877 and explains that the bug fix is limited to the stall retry, with range resume left for a later PR. The Problem section gives a reproduction path and the observed error. It does not follow the template's rule that the Verification section should avoid relying on partial checks. It also notes that some checks were not run (knip, Linux boot-service path). The template asks for the model and harness at the end, and the author included "Claude Opus 5.5 in Claude Code via T3 Code". The Scope section does not explicitly say why no issue is needed, but it links an issue, so that requirement is met.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between d8c7855 and e90ca3a.

Files selected for processing: 2

  • apps/server/src/cloud/pinnedRuntime.test.ts
  • apps/server/src/cloud/pinnedRuntime.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:S 10-29 changed lines (additions + deletions). vouch:unvouched Author is not yet trusted in the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants