Skip to content

fix(server): keep a dated snooze on awake threads; keep a scheduled model until a default turn takes it - #97

Merged
chrisohearn merged 4 commits into
integration/mcp-events-20261109from
r4/w4-tools-followups
Nov 13, 2026
Merged

chrisohearn merged 4 commits into
integration/mcp-events-20261109from
r4/w4-tools-followups

Conversation

@chrisohearn

@chrisohearn chrisohearn commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

Follow-up to #94 (ledger r4/w4-tools), from its delta review. This PR is stacked on #94's head c5992fb, which includes the integration tip a59003e. GitHub retargets it to integration/mcp-events-20261109 when #94 merges. Its diff is three commits.

In every acceptance check, <deploy_ms> is the deploy time in epoch milliseconds.

1. A default agent send no longer erases a dated snooze on a thread that counts as awake

Problem. A thread can count as awake while its snooze is still stored: a pending request, a waiting guest message, or a run that failed after the snooze. isSnoozed then returns false. #94 used that result for two decisions: whether to hold the message, and whether to end the snooze. So a default agent send (respectSnooze: true) to such a thread cleared the stored snooze for good. This is the same class as the incident: a dated hold ended by an agent that never meant to end it.

Change. The two decisions are now separate in Orchestrator.ts:

  • keepsSnooze: the sender respects the snooze, the message is not an automatic wake, the stored date is in the future, and it isn't a usage-limit snooze. When true, the stored snooze is never cleared.
  • respectsSnooze: keepsSnooze, the thread has run before, and isSnoozed is true. When true, the turn is held as before.

A thread that counts as awake, or has never run, gets the message at once and keeps its dated hold. t3_thread_send reports this as a delivered warning: "It counts as awake for now (a pending request or an error), so the message was delivered; the snooze stays."

Test. runtimeLayer.test.ts, "delivers to a snoozed thread that counts as awake without ending its snooze": a background wake fails after the snooze, then a respect-snooze send starts at once and the stored snoozedUntil stays 2099-01-01. On #94's head the snooze is cleared.

2. A scheduled model change stays until a default-model turn takes it

Problem. #94 cleared the schedule once no queued default-model run from before the change remained. When only explicit picks were left ahead, the change was dropped before any default turn used it. The explicit pick then ran and moved the thread to its model, so later default messages got the older pick instead of the scheduled model.

Change. threadAtRunStart now clears the schedule only in two cases:

  • a default-model turn starts and nothing queued before the change still waits, explicit picks included;
  • a turn starts that names a model after the change: an explicit pick queued later, or a new non-server message with modelSelection. That newer instruction replaces the change.

An explicit pick queued before the change runs with its own model and leaves the change pending. Server continuations that repeat an earlier run's model don't count as naming one.

Test. The scheduled-change case in runtimeLayer.test.ts now runs to the end: two default runs and one explicit pick queued before the change, and one default message after it.

  • The change survives the default runs while the pick still waits.
  • The pick starts with its own model, and the change stays pending.
  • The next default turn starts with the scheduled model and clears the change.

On #94's head the change is gone after the second boundary.

3. Test: a later completed run releases snooze-held turns only once the thread is awake

runtimeLayer.test.ts, "a later completed run releases snooze-held messages only once the thread is awake":

  • a background wake completes while the thread sleeps, and the held turn stays held and deferred;
  • after an unsnooze, the next completed run releases it, as a usage limit's resume does;
  • the released turn loses snoozeDeferred.

This is the path a usage-limit resume takes: handleRunUpdate runs releaseUnattendedHolds.

4. acknowledgeOnly refuses applyAt (P3 from the #94 resolution check)

configureAcknowledgeOnly refused modelSelection and runtimeMode but not applyAt. So {acknowledgeOnly: true, applyAt: "next_turn"} set the mode and silently dropped applyAt. It is now refused with invalid_request, and the configure test in core.test.ts has a refusal case for it.

5. A server continuation leaves a scheduled model change pending (P2 from the #97 review)

Problem. A restart continuation (RestartContinuation.ts) sends modelSelection: source.modelSelection with creationSource: "server". threadAtRunStart counted that as a default turn and cleared scheduledModelSelection, but the turn ran on the old model, so the scheduled change was lost. The next fork deploy restarts the server, which is exactly what triggers this.

Change. threadAtRunStart now classifies the starting turn as default, named or server. A server turn names a model from the server, on either start path (queued, identified by its message's creationSource, or immediate). It runs on its own model and leaves the thread, and so the schedule, unchanged. While a change is pending, queueTailModelSelection also skips a queued server continuation, so later default sends take the scheduled model rather than the repeated old one.

Test. runtimeLayer.test.ts, "keeps a scheduled model change through server continuations on the old model", covers both paths:

  • queued: a continuation behind the busy run starts on the old model, and the schedule survives; a default message queued after it carries the scheduled model and clears the schedule when it starts;
  • immediate: after the busy run ends, a continuation starts at once on the old model and keeps the schedule; the next default send starts on the scheduled model and clears it.

RestartContinuation.test.ts (50) still passes.

P3s, also done:

  • The delivered snooze warning now reads "It counts as awake for now (it never ran, or a request, guest message or error woke it), so the message went ahead as usual: started or queued. The snooze stays."
  • A new test, "keeps the snooze of a thread that never ran while delivering to it", covers the never-ran case. The pending-request case shares that path, and the error case is already tested.

Acceptance (read-only)

sqlite3 -readonly ~/.t3/userdata/statev2.sqlite "select thread_id, json_extract(payload_json,'$.snoozedUntil') from orchestration_v2_projection_threads where json_extract(payload_json,'$.snoozedUntil') > strftime('%Y-%m-%dT%H:%M:%fZ','now') and deleted_at is null"
sqlite3 -readonly ~/.t3/userdata/statev2.sqlite "select json_extract(payload_json,'$.scheduledModelSelection') from orchestration_v2_projection_threads where thread_id like 'mcp:launch:be52150e%'"

Pass:

  • threads with a dated snooze keep it across agent sends after deploy, with no thread.unsnoozed event whose actor is an MCP send with respectSnooze left at its default;
  • a schedule on the inbox clears only after a default-model run starts with it.

Notes

  • Docs: decision record 0017, docs/user/outside-agents.md and docs/orchestration-v2/orchestrator-mcp-server.md are updated for both changes.

  • Agent context: the new warning text is in a tool result, not the tool list, so the budget is unchanged. No migration.

  • Checks: under the resource rule, I ran only focused checks, under the shared lock with two workers:

    • the apps/server typecheck;
    • the five touched runtimeLayer cases (the two new tests, the scheduled change, the respect-snooze hold, and the post-only case);
    • OrchestratorMcpService, SnoozeDeferredDelivery, Orchestrator.control-reads, core and AgentContextBudget (158 tests).

    Hosted CI is the full gate.

🤖 Generated with Claude Code

@chatgpt-codex-connector

Copy link
Copy Markdown
Codex Review Summary: ❌ Failed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ❌ Failed 2026-11-13T04:57:12.838290Z
We couldn’t complete this review. Comment @codex review to try again.
a267c97 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

rallymewrx and others added 2 commits November 13, 2026 10:00
… model for default turns

From the PR #94 delta review:
- A default agent send to a thread that counted as awake only because of
  a pending request, guest message or error erased its dated snooze. It
  now delivers and keeps the stored snooze. So does a thread that has
  never run. The send's warning says it was delivered.
- A scheduled model change was cleared once only explicit picks were
  ahead of it, so a later default message got the older pick. It now
  stays until a default-model turn starts with it, after everything
  queued before it, and a model named after the change replaces it.
- Tests: a later completed run, such as a usage limit's resume,
  releases snooze-held turns only once the thread is awake.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
{acknowledgeOnly, applyAt} set the mode and silently dropped applyAt.
It is now refused like modelSelection and runtimeMode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chrisohearn
chrisohearn force-pushed the r4/w4-tools-followups branch from a267c97 to 4c746ef Compare November 13, 2026 17:01

@chrisohearn chrisohearn left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review of 4c746ef (review of record; commits on top of #94 head c5992fb)

Read-only Claude Opus review run by the round-four coordinator. Codex hosted review is at its usage limit, so this is the review of record.

What I checked

  • Read git diff c5992fb8c9 4c746ef5a6 and both commits, the PR body, and the surrounding Orchestrator.ts code: threadAtRunStart, queuedBeforeScheduledSelection, queueTailModelSelection, queuedRunModelSelection, queued-run start (~2569), direct turn start (~8087), the keepsSnooze/respectsSnooze send path (~7060-7110), releaseUnattendedHolds (3277) and queue.resume (15643).
  • Also read SnoozeState.isSnoozed, the SnoozeDeferredDelivery sweep, snoozeNotice.ts, snoozeOutcome, and configureAcknowledgeOnly.
  • Senders that set command.modelSelection: only RestartContinuation. MCP send, scheduled tasks, intake and the relay do not.
  • Dispatch and queued-run start both take threadDispatch.withLock, so a turn start and a scheduled-model take cannot interleave.
  • No tests run locally. Hosted CI is green on this exact head, including the Agent context budget. The budget is unchanged.

Checked and fine

  • Snooze split: an expired snooze and a usage-limit snooze still clear, and sends from the app still end a snooze. If a thread has a snooze and also counts as awake, a send keeps the stored snooze and delivers the message. The sweep releases held turns once the thread wakes. The new test exercises the keep branch.
  • Scheduled model, default branch: an explicit pick queued before the change keeps the change pending, and the next default turn takes it. Covered end to end. Threads without a scheduled change are unaffected.
  • acknowledgeOnly + applyAt is refused with invalid_request. The test fails without the fix.

P1: None.

P2: Orchestrator.ts:8087-8091 with RestartContinuation.ts:323: a restart continuation drops a scheduled model change without applying it. The continuation passes modelSelection: source.modelSelection with creationSource: "server". Server sends are excluded from namedModel, so the continuation takes the default branch. That branch clears the schedule, but the turn runs on command.modelSelection (the old model), not nextTurnModelSelection. Scenario: a change is scheduled behind busy run A, and the server restarts mid-A (as t3-service-update does). The continuation runs A's old model and clears scheduledModelSelection, so later default turns stay on the old model. Not a regression (c5992fb drops it the same way), but it contradicts the PR's claim that the change "stays until a default-model turn takes it". Fix: when a server send names a model, return projection.thread unchanged. Do the same on the queued path, where explicitModelSelection counts server runs as named.

P3

  • Orchestrator.ts:7080-7091 with SnoozeDeferredDelivery.ts: there is a small race. Turn A is snooze-held, and a wake fails, so the thread counts as awake. Within the sweep's 5-second tick, a respectSnooze send B starts directly and keeps the snooze. With B running, isSnoozed is true again, so A waits for the snooze date or a manual wake. A's contract ("held until it wakes") still holds.
  • snoozeNotice.ts:18 / OrchestratorMcpService.ts:3542-3546: the delivered warning names only "a pending request or an error". It also fires for threads that never ran, for guest messages, and when delivery is queued, where "delivered" is inaccurate.
  • Test gaps:
    • No test clears the schedule through namedModel.
    • The server-continuation exclusion is untested.
    • No test keeps the snooze for a thread that never ran, has a pending request, or has a guest message.
    • The notice text is untested.
  • Intended behaviour change: a snoozed thread that never ran now runs the message and stays snoozed.

Verdict: merge. The coordinator asks for the P2 fix in this PR before merging, since the next deploy restarts the server.

…nding

A restart continuation names the run's own model from the server. It was
treated as a default turn, so it cleared a scheduled change while running
on the old model, and the change was lost. Such a turn now runs on its
own model and leaves the schedule as it is. Later default sends skip it
as the queue tail while a change is pending.

The snooze notice's delivered text now also covers never-ran threads,
guest messages and queued delivery. Tests: a server continuation, queued
and started at once, keeps the schedule for the next default turn; a
never-ran snoozed thread keeps its snooze while a send is delivered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chrisohearn
chrisohearn changed the base branch from r4/mcp-tool-reliability to integration/mcp-events-20261109 November 13, 2026 17:19
@chrisohearn
chrisohearn changed the base branch from r4/mcp-tool-reliability to integration/mcp-events-20261109 November 13, 2026 17:19

@chrisohearn chrisohearn left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent delta review of 2437caa (review of record, follows 4c746ef review), plus a clean-merge check of e470b94

Read-only Claude Opus review run by the round-four coordinator. Codex hosted review is at its usage limit, so this is the review of record.

Merge check (coordinator, mechanical): e470b94 has parents 2437caa and integration tip e3f1747. git show --remerge-diff is empty, so there was no manual resolution. git diff 2437caa469 e470b9426d matches git diff c5992fb8c9 e3f174772c exactly: 7 files, 297 insertions, 15 deletions. That is #98, already reviewed.

What I checked

  • Read git diff 4c746ef5a6 2437caa469, the earlier review and the PR body. In Orchestrator.ts, read threadAtRunStart, queueTailModelSelection, queuedRunModelSelection, queuedRunFollowsThread, followsCurrentConversation, the queued start (2394-2584), the immediate start (8095-8125), and model choice at dispatch (7238, 7642, 7686).
  • Grepped every creationSource: "server" sender. Only three set modelSelection, so only they are classed "server": RestartContinuation.ts:323, EffectWorker.ts:353-357 (steer follow-ups and delegated completions), and ProviderFailoverDispatch.ts:48-51, which copies the original creationSource. The relay, scheduled tasks, intake, notify paths, UsageLimitRecoveryWorker, IdleCompactionService and QueuedSendCancelNotice set no model, so they stay "default" and still take the schedule.
  • Clients send "web" or "mobile", so a real user pick is "named".
  • The queued lookup is safe. queuedMessage is the run's userMessageId message. If it is missing, the start fails earlier at 2507. The command projection loads queued runs' messages, so serverRepeat sees them.
  • Traced the new test by reading the code: it fails on 4c746ef on both paths and passes on 2437caa. No tests run locally. Hosted CI on the exact head decides.

P1: None. P2: None.

P3

  • ProviderFailoverDispatch.ts:48-51 with Orchestrator.ts:8098-8106: a failover copies the original creationSource. A schedule pending behind a failed-over run survives when the original was a server message, but is cleared without being applied when it came from the web or MCP (as before). The same automatic action behaves two ways. Low impact.
  • Orchestrator.ts:2578-2584: a queued delegated completion that names a model is classed "server", but it follows the thread and gets the scheduled model, and the schedule stays pending until the next default turn. Harmless. But the claim that a server turn "runs on its own model" does not hold here.
  • Test gaps: no test clears the schedule through a named model, the notice text is untested, keeping the snooze for a pending request or guest message is untested, and failover and steer follow-up senders are not covered.

Earlier-review findings

  • P2, a restart continuation drops a scheduled change: resolved on the queued path (2578) and the immediate path (8098). queueTailModelSelection skips queued server repeats.
  • P3, snooze-sweep race: not addressed. Accepted, since "held until it wakes" still holds.
  • P3, delivered warning wording: resolved.
  • P3, test gaps: partly resolved (server continuation and never-ran snooze are covered).
  • Never-ran snoozed thread runs the message: intended, and now tested.

Verdict: merge (on green CI at e470b94)

@chrisohearn
chrisohearn merged commit bd06c88 into integration/mcp-events-20261109 Nov 13, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants