[Bug]: PR watch misses successful same-head CI reruns that start and finish between polls #34212

Before submitting

• I searched existing issues; related reports and their differences are listed below.
• I included a deterministic reproduction and live timing evidence.

Area

apps/server

Summary

A watched GitHub PR received its first checks-passed wake. After review, the agent marked the PR ready, which triggered another CI run on the same head SHA. The agent ended its turn explicitly awaiting that run. The new run started and succeeded entirely between two watcher sweeps. T3 subsequently read the final state but never woke the thread again.

The result is polling-phase dependent: if the evaluator observes the intermediate pending state it emits another completion wake; if it misses pending, the same new run's completion is suppressed.

Steps to reproduce

Live sequence

1. Use a GitHub Actions workflow triggered on both pull_request.opened and pull_request.ready_for_review, with a quick check such as validate.
2. Open a draft PR and call watch_pull_request. Let its initial checks pass and wait for the checks-passed notification.
3. Without pushing a commit, mark the PR ready. Have the agent end its turn awaiting the newly triggered CI run, keeping the existing watch active.
4. The new run must start and finish between two watcher reads. A slower runner that lets T3 observe pending can mask the problem.
5. Observe through the next full detail read. The snapshot becomes passing, but no completion notification or new agent run is created.

The live incident below was an existing authorized workflow, not a newly created public test PR. It was investigated using read-only GitHub queries, persisted watch events and server traces. No additional message, rewatch or runtime restart was injected into the measured interval.

Deterministic reproduction (no server, provider, runner or npm install required)

Requires a Node version exposing node:module.stripTypeScriptTypes (tested with Node 24.21.0). Save this as repro.mjs:

import assert from 'node:assert/strict';
import { readFileSync } from 'node:fs';
import { stripTypeScriptTypes } from 'node:module';

const source = readFileSync(process.argv[2], 'utf8');
// Fixtures never enter the DateTime-dependent no-checks branch.
const js = stripTypeScriptTypes(source).replace(
  'import * as DateTime from "effect/DateTime";', ''
);
const { evaluatePullRequestWatch: evaluate } = await import(
  'data:text/javascript;base64,' + Buffer.from(js).toString('base64')
);
const watch = {
  startedAt: '2026-11-11T00:00:00Z', headSha: 'same-head',
  failedChecks: [], passed: false, passedChecks: [], owedChecks: [],
  remarksThrough: '2026-11-11T00:00:00Z', remarkIds: [],
  conflicting: false, wakes: 0,
};
const results = [];
for (const required of [false, true]) {
  const detail = (status, run, headSha = 'same-head') => ({
    headSha, capabilities: { checksAlwaysReport: false },
    mergeability: 'mergeable', viewer: 'agent-user',
    author: { login: 'agent-user' },
    checks: [{
      name: 'validate', status, required, description: null,
      url: `https://ci.example/runs/${run}/jobs/validate`,
    }],
  });
  const first = evaluate(watch, detail('success', 'A'), [], 0);
  assert.equal(first.changes[0].kind, 'checks-passed');
  const sameRead = evaluate(first.next, detail('success', 'A'), [], 0);
  assert.deepEqual(sameRead.changes, []);
  const missed = evaluate(first.next, detail('success', 'B'), [], 0);
  assert.deepEqual(missed.changes, []); // Confirms the observed missing wake.
  const pending = evaluate(first.next, detail('pending', 'B'), [], 0);
  const control = evaluate(pending.next, detail('success', 'B'), [], 0);
  assert.equal(control.changes[0].kind, 'checks-passed');
  const pushed = evaluate(first.next, detail('success', 'B', 'new-head'), [], 0);
  assert.equal(pushed.changes[0].kind, 'checks-passed');
  results.push({
    required, initial: first.changes, sameRunReread: sameRead.changes,
    newRunWithoutPendingRead: missed.changes,
    newRunWithPendingRead: control.changes, newHead: pushed.changes,
  });
  if (process.argv.includes('--expect-wake')) {
    assert.ok(missed.changes.some(x => x.kind === 'checks-passed'),
      'A new completed CI run was not reported because pending was not observed');
  }
}
console.log(JSON.stringify(results, null, 2));

Download the actual release source and run:

curl -fsSL https://raw.githubusercontent.com/pingdotgg/t3code/bf469bb19669671d0df2efda74ed73867a7137c0/apps/server/src/orchestration-v2/pullRequestWatch.ts -o pullRequestWatch.ts
node repro.mjs pullRequestWatch.ts
# Explicit expected-behavior assertion, currently exits 1:
node repro.mjs pullRequestWatch.ts --expect-wake

The script strips TypeScript types and removes the external DateTime import. The fixtures never enter its no-checks branch. It runs the actual upstream evaluator; it does not reimplement the decision logic. The same missing-wake behavior was separately reproduced against the evaluator extracted from the installed binary.

Results for both an explicitly required gate and the fallback all-checks gate:

initial successful run A                     => checks-passed
reread of the same successful run A          => [] (correct deduplication)
new successful run B, pending not observed   => [] (missing completion)
new run B, pending observed before success   => checks-passed
successful run after a new head SHA          => checks-passed

Run B has the same check name but a different run URL. --expect-wake fails with:

AssertionError: A new completed CI run was not reported because pending was not observed

Expected behavior

An agent explicitly awaiting a newly started CI cycle should receive its completion even when the cycle finishes between polls. Repeated reads of the same completed run should remain deduplicated. At minimum, whether completion is reported should not depend on accidentally observing pending between two otherwise identical run histories.

This is not a request to merge automatically or to treat a wake as proof that merge/deploy policy is satisfied.

Actual behavior

The watch remains active, the PR snapshot updates to passing, and no new agent turn starts. The final agent message continues to say it is awaiting CI even though that CI has finished.

Diagnosis

The installed reactor uses a two-minute sweep interval. evaluatePullRequestWatch deduplicates success using the head SHA, passed, gate names and owed checks. It does not distinguish successive completed CI runs through their URL/ID/attempt.

With the same SHA and gate names, previously saved passed=true, and owedChecks=[], a second successful run does not satisfy the new-pass condition. A pending read resets passed, explaining the contrasting control result. No new gate names appear in this case, so the late-required-gate handling does not help.

A shorter interval may reduce frequency but cannot eliminate this race. Any fix should preserve same-run deduplication while recognizing a new awaited CI cycle; run identity may need to survive the provider/fingerprint/evaluator boundary.

Impact

Major degradation or frequent failure: an agent waiting for an authorized follow-up can remain idle indefinitely until a user supplies another message.

Version or commit

Live incident and installed evaluator: 0.1.8, release commit bf469bb19669671d0df2efda74ed73867a7137c0.

Deterministic public-source reproduction also confirmed against main at dea84724384fcc68ef802e7db8440d9abaf0ef26. This is a function-level check of main, not a full end-to-end run of that build.

Source: release evaluator, release reactor.

Environment

Linux ARM64 server, kernel 7.0.0-1013-oracle; T3 Code 0.1.8; Codex provider, gpt-6.1-sol; GitHub Actions. Local standalone reproduction: Node 24.21.0. No UI-specific behavior was tested.

Logs or stack traces

Sanitized live evidence; all times UTC on 2026-11-11:

06:07:54.000  Initial CI succeeds.
06:08:40.462  Watch enabled.
06:10:11.222  Initial checks-passed wake delivered; agent verifies first run.
06:15:28.454  Review completion wakes agent.
06:16:12.513  Watch sweep starts; new CI does not yet exist.
06:16:12.994  Watch sweep finishes.
              Agent marks draft PR ready, keeping head SHA unchanged.
06:16:29.000  New CI run created.
06:16:43.485  Agent turn ends, explicitly waiting for new CI.
06:16:45.779  PR snapshot updates: draft=false, checksState=pending.
              Saved watch still has passed=true and owedChecks=[].
06:17:18.000  New successful check completes.
06:17:19.000  New CI run completes successfully.
06:18:12.995  Next watch sweep starts.
06:18:13.428  Watch readGroup performs full detail read.
06:18:14.083  Sweep finishes successfully.
06:18:14.644  PR snapshot updates: checksState=passing.
              Watch remains active: passed=true, owedChecks=[].
06:56+        Observer confirms no new notification/run; no active run,
              no pending user request; GitHub reports PR CLEAN.

The snapshot synchronization and the watch evaluator use separate paths: a persisted pending snapshot does not mean that the periodic evaluator observed a pending gate. The actual watch sweeps bracket the complete new CI run.

Private PR URLs, thread IDs, account names and home paths are intentionally omitted. The synthetic reproduction above contains no organization-specific data.

Related issues

• [Bug]: GitLab PR watch misses manual-job completion between polls, leaving dependent work stalled #34110: closely related underlying timing problem, but GitLab optional manual jobs changing neutral to success; this report supplies a GitHub successful-run-to-successful-run case with captured watcher sweep timings. That report was closed as not planned after an initially inconclusive live reproduction; a later comment adds stronger evidence.
• [Bug]: PR watch reports "All N checks passed" again on the same head after a pending read or a rerun #32276: reports repeated pass notifications when pending is observed, and proposes pass-once-per-head semantics. This report demonstrates the complementary path: an agent awaiting a new run gets no completion when pending is missed. Please consider these together when deciding consistent rerun semantics.
• [Bug]: PR watch never reports a rerun that fails again, and on GitHub a failed run can be hidden by another event's run of the same check #32277: same-head reruns that fail again between reads; this report is about successful subsequent runs.
• [Bug]: watch_pull_request can miss late required gates that pass between polls #15362: a newly appearing required gate, already fixed; here the set of check names does not change.
• PR watcher misses successful replacement CI when job names change #30378: superseded cancelled checks with changed job names; here the replacement successful check names stay unchanged and no stale failure is needed for the deterministic reproduction.

Workaround

Manually prompt the waiting thread to verify the current run and continue. This supplies a manual wake but does not fix monitoring. No runtime patch, database write or service restart was performed for this investigation.