From 0aa21d210555c3cea19ef9a4b32195dead2cda48 Mon Sep 17 00:00:00 2001 From: "exe.dev user" Date: Fri, 30 Oct 2026 01:16:48 +0000 Subject: [PATCH] Document production Worker startup recovery --- docs/package-migration.md | 4 +++ docs/production-startup-recovery.md | 54 +++++++++++++++++++++++++++++ 2 files changed, 58 insertions(+) create mode 100644 docs/production-startup-recovery.md diff --git a/docs/package-migration.md b/docs/package-migration.md index 0847cc0..502ef0c 100644 --- a/docs/package-migration.md +++ b/docs/package-migration.md @@ -182,6 +182,10 @@ window may need a deliberate reload. Small root HTML is never restored from an o ## Rollback +If a current Worker startup failure blocks the live API prerequisite for a fix, +follow [production startup recovery](production-startup-recovery.md) to restore +a compatible working version before rerunning the normal deployment workflow. + Use **`ff4b51b`**, which retains all new APIs while restoring legacy page ownership. Never revert database state or rename/recreate a Worker to roll back the frontend. In an isolated anchor checkout, build legacy CSS; restore the last current/retained diff --git a/docs/production-startup-recovery.md b/docs/production-startup-recovery.md new file mode 100644 index 0000000..7d565a0 --- /dev/null +++ b/docs/production-startup-recovery.md @@ -0,0 +1,54 @@ +# Recovering a Worker startup failure + +The production workflow requires a healthy `/api/v1/session` response before +deployment. A Worker startup exception can therefore block deployment of its own +fix. Restore a compatible working Worker version first, then rerun the normal +workflow with every release gate enabled. + +## Recovery procedure + +1. Read the attached Reflection integration metadata and use the documented + Cloudflare API proxy. Start a temporary Worker tail and reproduce the failure + with read-only health and session requests. Record the exception and deployed + version ID; avoid saving request headers, bodies, or credentials. +2. Inspect the Worker's deployment history and version metadata. Select the most + recent compatible working version. Check its code, runtime, bindings, assets, + and Durable Object migration tag. A version created by a secret update before + a broken code deployment may retain the working code with the current secrets. + Do not force a rollback across changed secrets or incompatible state. +3. Use Cloudflare's [Create Worker Deployment API](https://developers.cloudflare.com/api/resources/workers/subresources/scripts/subresources/deployments/methods/create/) + to route 100% of requests to the selected version. This activates an existing + version; it does not revert database migrations or recreate state resources. +4. Verify `/healthz` returns `200` with `{"ok":true}` and an anonymous + `/api/v1/session` request returns `200` with `{"authenticated":false}` and + `X-Factorize-Contract: gen-2157-static-v1`. Run: + + ```sh + node scripts/verify-public-routing.mjs https://app.factorize.sh + ``` + +5. Rerun the failed production workflow for the merged fix. Monitor the API + prerequisite, asset retention, migrations, Worker deployment, and final public + verification. Verify the live version and anonymous browser loading again + after completion. Do not leave a temporary rollback as the final release. +6. Delete the temporary Worker tail and preserve the recovery evidence in a PR. + +## 2026-10-30 OAuth scope incident + +Cloudflare tails captured `TypeError: scopesSupported must not contain duplicate +values` on health, session, and internal trace-upload requests. `runs:write` had +become a canonical access scope but was also appended to the provider's legacy +scope list. OAuth provider construction threw before backend dispatch. + +[PR #317](https://github.com/GenerationSoftware/factorize/pull/317) deduplicated the +provider scope list, but its first deployment failed at the live session gate. +The failing production version was `ee5e4bf5-9b57-4182-b5ce-2c641d351d3e`. + +Recovery activated `aae2319a-c571-4be8-91e1-a3e7d1c4ef37`, whose script and runtime +matched the preceding working deployment and whose secrets were from the latest +pre-deploy secret update. Recovery deployment +`93089924-e637-4093-a9fa-c179fea8adc5` restored health and session responses at +01:14 UTC. Public routing/security/cache verification and an anonymous Chromium +page load passed, with no JavaScript page errors. The normal +[deployment workflow](https://github.com/GenerationSoftware/factorize/actions/runs/40984395531) +was then rerun for merged commit `24cf99c`.