Skip to content

[Bug]: Since #23522, a PR watch never wakes when checks pass between the watch call and its first read #36463

Closed as not planned

Description

@gperezmz

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

  1. Open a PR on GitHub whose checks take less than about 2 minutes (one required check, finishing in under a minute).
  2. While the checks are still pending, have the agent call watch_pull_request on it and end its turn, as the PR instructions tell it to.
  3. Let the checks pass before the next PullRequestWatchReactor sweep (it runs every 2 minutes).
  4. Wait.

Expected behavior

The thread wakes with "All 1 required check passed". The checks were pending when the watch started, so them passing is news.

Actual behavior

The thread never wakes. The first read after the watch starts sees the green checks, saves them as the baseline (passed: true), and sends no notification. Later reads find nothing new. The thread stays idle until the user steps in or the PR is merged by hand.

Timeline from one occurrence (UTC):

Time Event
23:59:44 watch_pull_request is called. The watch is created with headSha: null, passed: false, awaitingBaseline: "all". The checks are pending.
00:00:22 The last check passes.
00:00:33 First sweep reads the PR: saves passed: true, passedChecks: ["<required-check>"], awaitingBaseline is cleared, no wake.
00:02:35 Next sweep: no change, no wake.
00:04:37 The watch ends with wakes: 0, reads: 1 (the user merged by hand).

Cause

#23522 (fixing #22184) baselines the first read silently, so checks that were already done when the watch started aren't reported. But the baseline is taken at the first sweep, up to 2 minutes after the watch was created, not when watch_pull_request is called. In evaluatePullRequestWatch, a new watch has headSha: null, so headMoved is false and checksBaseline is true on that first read. That suppresses checks-passed even when the checks went from pending to passed after the watch started:

const headMoved = watch.headSha !== null && headSha !== watch.headSha;
const checksBaseline = watch.awaitingBaseline !== void 0 && !headMoved;
...
if (passedNow && (!passed || gateGrew)) {
  if (!checksBaseline) changes.push({ kind: "checks-passed", ... });

The tests added in #23522 cover checks that were green before the watch started. They don't cover checks that finish between the watch call and the first read. The same window also hides other changes that happen in that gap: a check that fails (checks-failed is behind the same !checksBaseline), a check that starts waiting for approval (checks-awaiting-approval, same guard), and a new conflict (awaitingBaseline === "all" suppresses conflicting). A failure is the worst of these, because the agent is never told CI broke.

How often you hit it depends on CI speed: any repo whose checks finish within about 2 minutes of the watch call can lose the wake.

Possible fixes

  • Take the baseline when the watch is created: read the PR in watch_pull_request (or right after it) and save its head and checks then.
  • Or, on the baseline read, only treat checks as already known if they finished before watch.startedAt (check runs have completed_at). Anything that finished after it is still news.

Impact

Major degradation or frequent failure

Version or commit

0.1.10-nightly.20261113.4656

Environment

Linux (Debian 13), T3 Code server, Claude provider (Opus 5.5), GitHub PR with a required check enforced by a repository ruleset.

Logs or stack traces

# saved watch state, from thread.pull-request-synced / thread.pull-request-link-updated events
23:59:44.611Z {"headSha":null,"passed":false,"passedChecks":[],"awaitingBaseline":"all","wakes":0}
00:00:35.732Z {"headSha":"<sha>","passed":true,"passedChecks":["<required-check>"],"owedChecks":[],"wakes":0,"checksPending":false}

# server trace
00:04:37 PullRequestWatchReactor.reportEnd "pull request watch ended"
  {"reason":"stopped","minutes":5,"quietMinutes":5,"wakes":0,"reads":1,"partial":false}

Workaround

None inside T3. The agent (or the user) has to check the PR by hand. #33662, if merged, would eventually rescue a stuck watch after 60 minutes.

Related

#22184, #23522, #30530 (same symptom, the watch never wakes, from a different trigger), #28183

Activity

  1. juliusmarminge commented on Nov 14, 2026

    @juliusmarminge
    Member

    I tried to reproduce this on the current build and could not.

    Commit 4bab2f3 (origin/main). Not seen in a live run: watch_pull_request is reachable only through a provider session's MCP token, and a live repro would mean timing a real GitHub PR's CI to finish inside the up-to-2-minute gap before the first PullRequestWatchReactor sweep. Instead I reproduced it through the real server path with a mocked host, in apps/server/src/orchestration-v2/runtimeLayer.test.ts. Real orchestrator and effect worker; PullRequestWatchService.setWatching (what the watch_pull_request MCP tool calls) is called while the host summary and detail report the required check 'test' as pending on head ad70160aaa; the check then finishes (success, or failure) before the reactor's first sweep; then reactor.sweep and worker.drain. Expected one notification ('#12: checks passed' or '#12: checks failed'); got none in both cases (expected [] to deeply equal ['#12: checks passed'], and the same for failed). Harness check: when the first sweep still sees the check pending and it finishes before a second sweep, both cases wake as expected, so the empty result comes from the first-read baseline. The code matches the reporter's analysis: Orchestrator.ts creates the watch with headSha null and awaitingBaseline 'all'; setWatching does a fresh read at watch time (summary with bypassCache, which includes headSha and checksState) only to refuse merged PRs and throws the state away; evaluatePullRequestWatch then has headMoved false and checksBaseline true on the first read, which hides checks-passed, checks-failed and checks-awaiting-approval, and awaitingBaseline 'all' hides conflicting.

    Closing as not planned. If it still happens for you, reply with the exact steps and the version you are on and it will be looked at again.

  2. keeperxy commented on Nov 15, 2026

    @keeperxy
    Contributor

    Note

    🤖 Claude Opus 5.5 responding on behalf of @keeperxy

    @juliusmarminge This still happens, in a live run against real GitHub CI, twice in one thread. You asked for steps and a version, so here they are, plus the evidence from the server.

    Version: 0.1.10-nightly.20261114.4709 (built from dfd2187ccb, published 2026-11-14T23:19Z). That build contains 4bab2f3ae7, the commit from your no-repro comment, and the server ran it for the whole session (process started 2026-11-14T23:27Z). The baseline logic is unchanged on current main (32c069eed1). The only later commits to these files change comment-text quoting and GitLab rebase wakes.

    Setup: self-hosted T3 Code server on Linux, Codex provider. A private GitHub repo with GitHub Actions checks (unit-and-matrix, e2e) plus Vercel. No required checks are configured, so the gate is every check. e2e takes about 1m40s.

    Steps:

    1. The agent opens a PR. CI starts.
    2. While e2e is still in_progress, the agent calls watch_pull_request. It returns watching: true, wasWatching: false, and the agent ends its turn, as the PR instructions say.
    3. e2e passes a few seconds later, before the first PullRequestWatchReactor sweep.
    4. Nothing happens. The thread stays idle until the user asks for status, and the PR is merged by hand.

    Timeline (UTC)

    Check times come from the GitHub check-runs API. Watch times come from server.trace.ndjson (orchestrationV2.dispatch.withReceipt spans for thread.pull-request.watch and thread.pull-request-watch.sync).

    PR A

    Time Event
    04:54:24 unit-and-matrix completed, success
    04:54:27 e2e started
    04:56:03.7 watch_pull_request → thread.pull-request.watch (e2e still in progress)
    04:56:08 e2e completed, success. Every check is now green
    04:56:12 the agent's turn ends
    04:57:41.7 first thread.pull-request-watch.sync: baseline saved as passed: true, no wake
    07:08:10 the user asks for status (2h12m idle)
    07:10:28 watch ends: reason: "merged", wakes: 0, reads: 5

    PR B (same thread, about 2 hours later)

    Time Event
    07:12:16 unit-and-matrix completed, success
    07:12:19 e2e started
    07:13:50.7 watch_pull_request → thread.pull-request.watch (e2e still in progress)
    07:14:00 e2e completed, success
    07:14:02 the agent's turn ends
    07:14:29.9 first thread.pull-request-watch.sync: baseline saved as passed: true, no wake
    07:52:47 the user asks why nothing happens (38m idle)
    07:54:42 watch ends: reason: "merged", wakes: 0, reads: 2

    In both cases the checks went green 5s and 10s after the watch call, and the first read came 98s and 39s after it. That is the gap your runtimeLayer.test.ts run hit.

    Server log (PullRequestWatchReactor.reportEnd)

    {"message":"pull request watch ended","pullRequest":"github.com/<owner>/<repo>#A","reason":"merged","minutes":134,"quietMinutes":134,"longestQuietMinutes":134,"wakes":0,"reads":5,"partial":false}
    {"message":"pull request watch ended","pullRequest":"github.com/<owner>/<repo>#B","reason":"merged","minutes":41,"quietMinutes":41,"longestQuietMinutes":41,"wakes":0,"reads":2,"partial":false}

    Final saved watch state (endedWatch in the thread projection)

    // PR A
    {"startedAt":"2026-11-15T04:56:03.755Z","headSha":"<sha A>","failedChecks":[],"passed":true,
     "passedChecks":["unit-and-matrix","e2e","Vercel","Vercel Preview Comments"],"owedChecks":[],
     "awaitingApproval":[],"approvalAnnounced":false,"conflicting":false,"mergeQueueRemovalId":null,
     "wakes":0,"checksAwaitedSince":null,"checksPending":false}
    // PR B
    {"startedAt":"2026-11-15T07:13:50.672Z","headSha":"<sha B>","failedChecks":[],"passed":true,
     "passedChecks":["Deploy Convex Preview","unit-and-matrix","e2e","Vercel"],"owedChecks":[],
     "awaitingApproval":[],"approvalAnnounced":false,"conflicting":false,"mergeQueueRemovalId":null,
     "wakes":0,"checksAwaitedSince":null,"checksPending":false}

    passed: true with wakes: 0 and no awaitingBaseline: the baseline took in the pending → passed transition.

    Why (confirmed against current main)

    • PullRequestWatchService.ts#L126: setWatching reads a fresh summary (bypassCache: true, with head and checksState) at watch time. It uses that read only to reject merged PRs and then drops it.
    • The watch is created with headSha: null, awaitingBaseline: "all". The first read happens only at the next sweep (SWEEP_MINUTES = 2). A new watch has no change feed until that sweep adds it to currentGroups.
    • On that first read, checksBaseline is true, so checks-passed, checks-failed and checks-awaiting-approval are suppressed, and awaitingBaseline === "all" also hides conflicting.

    So any CI that finishes within about 2 minutes of the watch call loses its wake. A failure is lost the same way, so the agent never learns that CI broke.

    Your own run in the closing comment shows the same result (expected [] to deeply equal ['#12: checks passed'], and the same for failed). The factory:no-repro close looks like a misfire. Could you reopen this?

    Either fix from the issue would cover both of our cases:

    1. Save the head and checks state from the read setWatching already makes as the baseline. Then "pending at watch time" is known, and the first sweep reports the transition.
    2. Or, on the baseline read, treat only checks with completed_at < watch.startedAt as already known.

    If you need anything else, such as the full trace spans or a test against a specific commit, reply here. We are watching this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions