Skip to content

feat(kix): recover jobs from lost and blocked publications - #190

Merged
kixelated merged 5 commits into
mainfrom
quest/m0/kix/recovery
Oct 22, 2026
Merged

kixelated merged 5 commits into
mainfrom
quest/m0/kix/recovery

Conversation

@kixelated

@kixelated kixelated commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Finishes quest/m0/kix/recovery.md. A maintainer can get a failed or blocked kix job moving again from the inbox without going to GitHub, and a retry keeps answers that were already given.

Changes

  • Retry continuity. createJob now records which failed job a retry continues (jobs.continues, migration 0016_recovery.sql). store.decisions follows that chain instead of matching jobs by pull request, so a failed planning job that pushed commits but never opened a PR keeps its answered rounds in the retry's decision record.
  • Unrecorded pull requests.
    • Authorizing the retry: continued() in state.ts gives the PR a retry continues. That is the PR the failed job recorded, or else the only open observed PR on the job's branch whose head is a commit the job published (head or pending). Matching the commit rather than just the branch name means a contributor's PR on a same-named branch still claims the quest, which was the concern raised on kixelated/quest#171. The inbox's Try again button and the start route both use it.
    • Delivery: deliver adopts the branch's only open PR from the service fork, so it no longer needs the PR to have been created after the current job. Publication already checks that the branch is where the job left it. A PR opened since the job and then closed by a maintainer is still not replaced.
  • Start over. A publication decision caused by a service fork branch that changed outside its job, or was left by earlier work (BranchChanged), now has restart: true and shows a Start over button. The button can also be called through the answer route as restart: true. Starting over rechecks the answering maintainer's authority, then claims the restart in D1 (store.restart): it answers the decision, marks the old job cancelled, and queues a new job at the job's revision with a fresh session and no recorded PR. Only after that claim succeeds does it force-reset the fork branch to the revision (Branch.reset), so a concurrent answer or cancel refuses the restart before any commit is discarded. If the reset fails, store.hold puts the replacement, which hasn't run yet, on the same Start over decision so the maintainer can retry. If a runner already took the replacement, it finds the branch changed when it publishes. If the branch's PR is still open, the new job's delivery adopts it. An iterated PR's head is never offered for reset.
  • Seeded fake GitHub (cloudflare/dev/github.ts, dev/seed.ts): now models the service fork (its identity and forced or created branch refs) and PRs opened from the fork. The seed adds a failed job whose PR went unrecorded and a waiting job whose branch changed, so both states show up on the local server.
  • Docs: cloudflare/GITHUB.md and cloudflare/README.md. The quest file and its link are removed.

Validation

  • just check and just test pass after merging main (Worker: 319 tests).
  • New seed.test.ts case, run through the seeded fake GitHub and the inbox forms: Try again on a quest whose failed job's PR fix(skills): route every follow-up through /quest-plan in the foreground #20 went unrecorded, and Start over resetting the fork branch and queueing a new job. Repeating it is refused.
  • Start over race regressions in seed.test.ts, also through the seeded fake GitHub: a cancel injected while the restart rechecks GitHub leaves the fork branch untouched, and a failed reset leaves the replacement waiting on Start over, which then succeeds. Both fail with the earlier reset-then-claim ordering.
  • New publish.test.ts cases: decisions survive a chain of two failed jobs with no PR, the unrecorded open PR is adopted on retry, Start over followed by republishing keeps the PR, and an iterated head is never offered Start over.

Limitations

  • I haven't checked on real GitHub whether a PR stays open when its head branch is force-reset to the base. The design works either way: the new job adopts the PR if it's still open and opens a new one if it isn't.
  • Delivery adoption is tested with the stateful fake in publish.test.ts. The seeded fake doesn't model commits or PR creation, so it covers only the inbox-level paths.

Decisions

Settled by the maintainer:

  1. Start over resets the fork branch to the job's original revision, not the current default-branch commit. It's deterministic, and a retry or a new start already picks up the current commit.
  2. Start over replaces the job instead of requeueing it. The old job is marked cancelled and a new one is queued with no continues link, so the old answers drop out of the decision record because their work was discarded. Requeueing would keep the old createdAt, and a PR closed by the reset would then block delivery.
  3. Keep the two GitHub fakes for now. dev/github.ts covers only what the inbox flows need, and the publication internals stay on publish.test.ts's own fake. Merging them is a suggested follow-up below.

Chosen while addressing review, open to change:

  1. A failed reset holds the replacement job rather than rolling back the claim. The replacement waits on the same Start over decision. Undoing the claim would mean moving a cancelled job back to waiting, and job states only move forward. If the Worker stops between the claim and the reset, the replacement runs and then finds the branch changed when it publishes. That costs one agent run but discards nothing.
  2. The recovery migration is now 0016_recovery.sql, after main's 0015_activity_reads.sql. The recovery seeds also moved off main's new PR feat(skills): rename quest-complete to quest-finish #17 and themes draft: the unrecorded PR is now fix(skills): route every follow-up through /quest-plan in the foreground #20, and the blocked job is on a new quest/m1/fonts.md.

Suggested follow-ups

  • Merge test/publish.test.ts's stateful GitHub fake into cloudflare/dev/github.ts (commits, fast-forward refs, PR creation), so the local server can run publication end to end. Recommended.
  • Verify on a pilot installation what happens to a PR when its head is reset to the base, as part of Operator setup. Recommended, low effort.
  • Show a restarted job's predecessor and its answers in the inbox history. Optional.

🤖 Generated with Claude Code

(written by Claude Opus 5.5)

kixelated and others added 2 commits October 21, 2026 22:39
Quest-Claim: f5cf5caa948c10bc
A retry records the failed job it continues, and a planning PR's decision
record follows that chain, so answers survive a failure that opened no PR.

A failed job's PR whose number went unrecorded no longer blocks its retry:
the inbox and the start route accept the only open PR observed on the job's
branch at a commit it published, and delivery adopts the branch's only open
PR.

A publication decision on a service fork branch that changed outside its job,
or was left by earlier work, offers Start over: kix resets the branch to the
job's revision and queues a fresh job there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Recover jobs from lost and blocked publications feat(kix): recover jobs from lost and blocked publications Oct 22, 2026
@kixelated

Copy link
Copy Markdown
Collaborator Author

Implementation pushed in cefdbbd. just check and just test pass locally. This PR stays a draft: three implementation choices are waiting on a maintainer, listed under Decisions needed in the description. Each has a recommendation; none blocks the code.

(written by Claude Opus 5.5)

@kixelated
kixelated marked this pull request as ready for review October 22, 2026 15:52
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-22T15:55:00.943304Z 3461982 Draft marked ready
🔒 Security Review ✅ Completed 2026-10-22T15:56:39.036782Z 3461982 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3461982ad9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread cloudflare/src/jobs/publish.ts Resolved
kixelated and others added 2 commits October 22, 2026 09:19
Start over force-reset the service fork branch before recording the
restart in D1, so a concurrent answer or cancel could let the reset
discard commits and then refuse the restart. Replace the job first, then
reset. If the reset fails, the replacement waits on the same Start over
decision instead of running against the unreset branch.

Regression tests drive both cases through the seeded fake GitHub.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Renumber the recovery migration to 0016 after main's 0015_activity_reads,
and move the recovery seeds off main's new pull request 17 and themes
draft: the unrecorded pull request becomes 20, and the blocked job moves
to a new fonts quest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Merging via /quest-merge at deba914.

  • Changes: retry continuity (jobs.continues, migration 0016_recovery.sql), adoption of an unrecorded open PR on retry and delivery, and a Start over action for BranchChanged publication decisions that claims the restart in D1 before force-resetting the service fork branch.
  • Review: the Codex finding (reset before claim) is fixed in 4cad6d7 with regression tests; its thread is resolved.
  • Decisions: the maintainer keeps holding the replacement job on a failed reset rather than rolling back the claim. The other settled decisions are in the PR body.
  • Checks: all required checks pass. The PR is mergeable with no conflicts against main. quest check --base origin/main passes on the head. Main has no other 0016_* migration.

(written by Claude Opus 5.5)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant