Skip to content

fix(ci): let worker-live ride out a rolling deploy instead of failing it - #102

Merged
WomB0ComB0 merged 1 commit into
mainfrom
fix/worker-live-rollout
Oct 3, 2026
Merged

WomB0ComB0 merged 1 commit into
mainfrom
fix/worker-live-rollout

Conversation

@WomB0ComB0

Copy link
Copy Markdown
Member

Summary

worker-live failed on the v0.4.7 pin bump (#101) although the deploy was correct:

attempt 5: live v0.4.6 @ 9bac29b8, waiting for v0.4.7 @ 10a74187
  HEAD content-length 40084 matches the body
::error::install.sh: served bytes hash to 6cfe3be1… (v0.4.6), expected 08620772… (v0.4.7)

On attempt 6 the manifest flipped to v0.4.7, so the poll loop exited. The very next GET /install.sh reached an edge still running v0.4.6, and the content check failed at once. Workers deploys roll out gradually; Workers Builds finished at 23:04:12 and the check gave up seconds later. A re-run passed: serves v0.4.7 @ 10a74187, 7 artifacts verified.

Change

The content checks (HEAD/GET length, per-route digests, root route) now run as one pass, retried within the existing deadline. Each response's x-resq-commit decides what a mismatch means:

response meaning action
stamped with another commit rollout lag retry the pass; at the deadline fail as "has not reached every edge", worded as timing
stamped with main's commit, wrong bytes wrong deploy fail immediately, as before
no x-resq-commit unattributable fail immediately
request throws (timeout, reset) transient retry within the deadline

HEAD and GET of /install.sh must also both come from main's deploy before their lengths are compared, because the same race could otherwise compare two versions' lengths. The manifest poll and the deadline itself are unchanged.

Test plan

The workflow's own script was extracted (ORIGIN and timers overridden) and run against a local mock serving real Worker responses: main's routes for "new", and the Worker's version-locked /v0.4.6/... routes for "old", so the old responses carry v0.4.6's real commit and bytes.

scenario new script old script
clean deploy pass, 7 artifacts —
rollout race (first 3 content requests old) 2 retries, then pass fails with the production error, byte for byte
wrong deploy (main's commit, tampered bytes) fails immediately —
stuck rollout (always old) retries to the deadline, then the "not reached every edge" failure —
missing x-resq-commit fails immediately —
  • actionlint and zizmor clean
  • CI
  • The next deploy's worker-live run

The v0.4.7 deploy failed worker-live although it was correct: the manifest already read v0.4.7, and the next GET /install.sh reached an edge still serving v0.4.6, so the content check failed at once. Workers deploys roll out gradually; the re-run passed.

Content checks now run as one pass retried within the existing deadline, keyed on each response's x-resq-commit. A response from another commit is rollout lag and retries; one from main's commit with wrong bytes, or with no x-resq-commit, still fails immediately. HEAD and GET of install.sh must both come from main's deploy before their lengths are compared.
@WomB0ComB0
WomB0ComB0 requested a review from a team as a code owner October 3, 2026 23:12
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
get-resq-software 1aebfeb Oct 03 2026, 11:11 PM

@github-actions github-actions Bot added C-Bug Something isn't working size/L PR size: large (100-499 lines changed) labels Oct 3, 2026
@coderabbitai

coderabbitai Bot commented Oct 3, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 37 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 1d89f59f-b9f4-4081-9f79-be5985b26610
📥 Commits

Reviewing files that changed from the base of the PR and between 54ac117 and 1aebfeb.

📒 Files selected for processing (1)
  • .github/workflows/worker-live.yml
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added the A-DevOps CI, workflows, actions, and repo infra (.github) label Oct 3, 2026
@github-actions

github-actions Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

This pull request significantly improves the deployment verification workflow for the live worker. The previous implementation was susceptible to false negatives during Cloudflare Workers gradual rollouts. This change introduces a more robust polling and verification mechanism that accounts for this "eventual consistency" by checking commit hashes and retrying on mismatches until a deadline is reached.

After a thorough review, I found no security vulnerabilities, logic bugs, or performance issues. The changes are well-thought-out and improve the reliability of the deployment pipeline. Therefore, I approve of these changes.

Generated by ai-auditor for #102 · gem25pro · 23.2 AIC · ⌖ 3.96 AIC · ⊞ 12.8K · ◷

@WomB0ComB0
WomB0ComB0 merged commit b52d7a7 into main Oct 3, 2026
32 checks passed
@WomB0ComB0
WomB0ComB0 deleted the fix/worker-live-rollout branch October 3, 2026 23:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-DevOps CI, workflows, actions, and repo infra (.github) C-Bug Something isn't working size/L PR size: large (100-499 lines changed)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants