Skip to content

fix(downloads): serve installers from ScientFactory's mirror and never dead-end a download - #74

Merged
yaacovcorcos merged 4 commits into
mainfrom
fix/download-release-resilience-20261113
Nov 13, 2026
Merged

yaacovcorcos merged 4 commits into
mainfrom
fix/download-release-resilience-20261113

Conversation

@yaacovcorcos

Copy link
Copy Markdown
Contributor

Problem

/api/download/:asset resolved the release by fetching GitHub's latest handoff whenever the data center's five-minute edge cache was cold. Any refusal, error or timeout became release_fetch/upstream_unavailable and a bare JSON 503 with no way forward.

Production: 15 download_failed events (Oct 15 – Nov 12); on Nov 12, 7 failures vs 17 clicks.

Root cause (agreed independently by Claude and Codex)

Proven: the click path depended on a live, unauthenticated GitHub request on every cold cache, with no durable fallback and no recovery link. Pages Functions invocation analytics for the failure bursts (Nov 12 00:41:37, 05:23:54; Nov 10 14:22:39) show each failed request finished in ~0.3 s with two subrequests. These were fast refusals or errors from GitHub, not timeouts. They arrived as bursts of simultaneous requests against an expired cache, where some requests failed and others in the same burst succeeded.

Not proven: which GitHub response it was. The status and error were discarded, and Pages logs do not persist. 30-way concurrent fetches from Cloudflare's edge (TLV) and local curl all succeeded today. Codex rates the specific upstream cause low-confidence. This fix removes the dependency instead of tuning around one guessed status, and it records the status from now on.

Change

  • Mirror worker (workers/release-mirror, scientfactory-release-mirror, cron every 10 min):

    • Copies the four Stable installers from the GitHub release into the existing scient-downloads R2 bucket (downloads.scientfactory.com), at scient-desktop/<tag>/<name>.
    • Each write carries the handoff's SHA-256, which R2 verifies.
    • Writes scient-desktop/stable.json last. A failed run leaves the previous pointer.
  • Resolution order:

    1. Edge cache.
    2. Mirror, if confirmed within 6 h.
    3. GitHub: one retry for no-response/timeout/5xx, none for 4xx including 403/429; 4 s per attempt.
    4. The mirror however old.

    All sources pass parseReleaseHandoff.

  • No dead end: if nothing resolves, the response is a 302 to github.com/ScientFactory/scient-desktop/releases/latest. download_failed now records upstream_status, upstream_error, upstream_elapsed_ms, upstream_attempts and fallback.

  • Destination policy: besides the canonical GitHub path, it accepts only https://downloads.scientfactory.com/scient-desktop/v<semver>/<installer>, with no query string.

  • Pages: binds DESKTOP_DOWNLOADS to scient-downloads in wrangler.jsonc.

  • Docs: docs/architecture/desktop-downloads.md.

Evidence

  • bun run check passes: format, typecheck, events and mirror types and typecheck, 179 tests, build, edge build.
  • Live mirror run on Cloudflare (wrangler dev --remote --test-scheduled, against a temporary bucket that has since been deleted):
    • First run copied all four v0.6.28 installers in 49.7 s and promoted the pointer.
    • Second run copied nothing and did not re-promote (1.1 s).
    • The mirrored Scient-0.6.28-x64.exe SHA-256 a9783a37…afb07c equals the handoff's digest.

Deployment

  1. Merge: Pages deploys with the binding. Until the mirror is populated, behavior is GitHub with retry plus the release-page fallback.
  2. bun run mirror:deploy deploys the worker. Its first cron populates scient-desktop/ in scient-downloads.
  3. Verify that HEAD /api/download/<asset> returns 302 to downloads.scientfactory.com once the 5-minute edge cache turns over.

Rollback: revert the PR. The worker can be deleted; mirrored objects are inert.

Owner authorized merge in the conversation (investigate with Codex, fix, Codex review, then merge).

Documentation impact: Updated — docs/architecture/desktop-downloads.md

…r dead-end a download

A download click resolved the release by asking GitHub for the handoff whenever the
data center's five-minute cache was cold. Any refusal or error ended in a JSON 503.
Production failures came in ~0.3 s bursts of simultaneous requests against an expired
cache; GitHub's status was never recorded.

- The new scientfactory-release-mirror worker copies each Stable installer into the
  scient-downloads R2 bucket (downloads.scientfactory.com) every ten minutes. R2 verifies
  the handoff's SHA-256 on write; the Stable pointer is written last.
- Downloads use the mirror while the worker has confirmed it within six hours, then
  GitHub (one retry for timeouts, network errors and 5xx; none for 4xx), then the mirror
  however old.
- When nothing resolves, the visitor is redirected to the GitHub latest-release page
  instead of a 503, and download_failed records GitHub's status or error name, elapsed
  time, attempts and the fallback.
- The trusted destination policy accepts the mirror's versioned installer paths only.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Deploying scientfactory-website with  Cloudflare Pages  Cloudflare Pages

Latest commit: 249a026
Status: ✅  Deploy successful!
Preview URL: https://8c2cc81b.scientfactory-website.pages.dev
Branch Preview URL: https://fix-download-release-resilie.scientfactory-website.pages.dev

View logs

…er only over what this run saw

Review findings on #74:
- An installer key already holding other bytes is refused instead of overwritten, and a new
  copy is written with If-None-Match: *, so published bytes (and year-long caches of them)
  never change under a tag.
- The Stable pointer is written with a compare-and-swap on the version this run observed, so
  a slower run that saw an older release cannot replace a newer promotion.
A run whose handoff answer was delayed past another run's promotion read the pointer etag
afterwards, so its compare-and-swap succeeded and rolled the pointer back. The etag is now
captured first; the regression pauses the older run's handoff response.
@yaacovcorcos
yaacovcorcos merged commit 9a48d24 into main Nov 13, 2026
2 checks passed
@yaacovcorcos
yaacovcorcos deleted the fix/download-release-resilience-20261113 branch November 13, 2026 09:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant