[Bug]: PR watch never wakes again after an early check failure, so threads wait forever #30530

Activity

1. aglowienka commented on Nov 14, 2026

   

   aglowienka

   on Nov 14, 2026

   More actions

   Note

   Written by Claude Fable 5.2 on behalf of Adam

   The same bug happened today on 0.1.9, the latest stable release. It was a live GitHub PR with real CI and the normal two-minute sweep. This issue describes a repository with no required checks. In this incident the check that failed first is a required check, so both variants are confirmed.

   Environment

   • T3 Code Desktop 0.1.9 on macOS arm64, Darwin 27.2.0, with its local server 0.1.9.
   • Provider: Claude, model claude-fable-5-2.
   • Host: GitHub. 7 of the 12 checks on the PR are required.

   What happened

   All times are UTC on 2026-11-14. They come from the thread history, server.trace.ndjson and the check run timestamps of GitHub.

   Time	Event
   08:54:23	[C2] Check labels fails on the new head. It is a required check that fails until a person adds a label.
   08:55:22	The sweep reads the PR. Three required [T1] checks are still running. The thread gets the wake "checks failed, new comments".
   08:55:38	The agent ends its turn. Its last message says that the result of the three running checks will wake it, and that it will then trigger a review.
   08:55:29 to 08:55:47	The remaining checks finish. All of them pass.
   08:57:24	The next sweep reads the PR and sends no wake.
   08:59 to 09:07	Five more sweeps exit with Success and send no wake.

   The watch stayed on. The user reported that the thread still hangs although the checks are complete.

   Evidence

   # server.trace.ndjson, PullRequestWatchReactor spans, all exit Success
   08:55:22Z  sweep 2027ms  readGroup 1460ms  record 19ms    <- wake "checks failed, new comments"
   08:57:24Z  sweep 2059ms  readGroup 1040ms  no record span  <- all checks finished 97 seconds earlier
   08:59:26Z  sweep  559ms  readGroup    0ms
   09:01:27Z  sweep  577ms  readGroup    0ms
   09:03:28Z  sweep  521ms  readGroup    0ms
   09:05:28Z  sweep  598ms  readGroup    0ms
   09:07:29Z  sweep  706ms  readGroup    0ms
   
   # stored watch, orchestration_v2_projection_threads.payload_json, read after the 09:07 sweep
   "failedChecks": ["[C2] Check labels"], "passed": false, "passedChecks": [],
   "owedChecks": [], "awaitingApproval": [], "conflicting": false, "wakes": 0
   snapshot.checksState: "failing", mergeability: "mergeable", state: "open"
   
   # gh pr checks --required
   [C2] Check labels                fail  6s
   [B2] Build amd64 Docker images   pass  30s
   [B2] Build arm64 Docker images   pass  43s
   [T1] Behat                       pass  1m3s
   [T1] Clean code                  pass  57s
   [T1] OpenAPI validation          pass  13s
   [T1] PHPUnit                     pass  48s
   

   Evaluator replay with the data of this incident

   I imported the unmodified apps/server/src/orchestration-v2/pullRequestWatch.ts from four revisions. I called evaluatePullRequestWatch with the stored watch and the real check runs of this PR. The only stub is effect/DateTime, which the evaluator calls only when the head moves. I rebuilt the check list of the 08:55:22 sweep from the startedAt and completedAt values of GitHub.

   The table shows the changes that each revision returns.

   Read	Checks	v0.1.9 at 24e77473ec	main at 2fa77fbf55	#28020 at 9c0ae214c7	#30531 at cd94f5c0e6
   Sweep at 08:55:22	3 pending, 1 failed	checks-failed	checks-failed	checks-failed	checks-failed
   Sweep at 08:57:24	0 pending, 1 failed	none	none	checks-finished	checks-failed
   Next sweep	0 pending, 1 failed	none	none	none	none
   Control with [C2] Check labels passing	0 pending, 0 failed	checks-passed	checks-passed	checks-passed	checks-passed

   So the bug is present on 0.1.9 and on main from 2026-11-14T09:08Z. Both open fix PRs send exactly one more wake for this required-check case. I tested only the evaluator of the two PR heads. I did not run a server built from them.

   One more result applies to both PRs. I also replayed the watch as 0.1.9 stored it, with owedChecks: [], against the finished checks. Both PR evaluators return no changes there. A watch that already missed the wake before an upgrade stays silent for that head.

   What #33214 changes

   #33214 is on main and not in 0.1.9. On main the same read that returns no changes also changes checksPending from true to false. According to the description of #33214, the thread then leaves Working and returns to the inbox. I did not run a nightly, so I have not confirmed that in the UI.

   The agent still gets no wake. In this incident the agent told the user that it would act when the remaining checks finish. With #33214 the thread would look finished, and that action would still never happen.

   Steps to reproduce the required-check variant

   1. Use a repository with a required check that fails within seconds and stays failed until a person acts. Here it is a label check.
   2. Make sure that other required checks run for at least one more sweep. Here they run for about 90 seconds.
   3. Open a PR and ask the agent in a thread to watch it.
   4. Wait for the "checks failed" wake and let the agent end its turn.
   5. Wait until the remaining checks pass.
   6. Wait for the next sweep.

   Expected: one wake that says all checks have finished and names the check that is still failed.
   Actual: no wake.

   Related

   • watch_pull_request never wakes when the remaining checks finish after a required check failed #25497 is the earlier report of the required-check variant. It is closed, and this issue now tracks the missing wake.
   • PR watch never wakes the agent when checks finish after an early failure #28019 reported the variant without required checks. It was closed as a duplicate of watch_pull_request never wakes when the remaining checks finish after a required check failed #25497.

   No source patch, database write or server restart was applied. Investigated through T3 Code triage.