Skip to content

A queued delegated-task completion reverts a model change made while it waited #22577

Closed

Description

@f4llenz

What happened

An agent delegated a task with delegate_task. The child finished while the parent thread was busy with another turn, so its completion notice was queued. While the notice waited, the thread's model was changed with t3_thread_configure. The call succeeded. When the notice's turn started, the thread's model switched back to the earlier one. t3_thread_configuration then read back the old model, and that turn and the turns after it ran on the old model.

It happened twice in a row on the same thread. A model change made before a notice was queued held.

Diagnosis

The queued completion run starts on a copy of the thread's selection taken when the notice was queued. If that copy differs from the thread's selection, the start writes the copy back to the thread.

  1. ProviderContinuationService queues the completion notice with queue_after_active and no modelSelection (ProviderContinuationService.ts#L119-L135).
  2. message.dispatch falls back to the thread's selection (Orchestrator.ts#L4983), and the queued run stores that copy.
  3. When the queue starts the run, queuedRunModelSelection returns the stored copy for everything except PR-watch monitor updates. Since fix(server): queued PR updates follow the takeover provider #21314, those follow the thread's current selection (Orchestrator.ts#L655-L672).
  4. The start compares the copy with the thread's current selection. When they differ, it emits thread.model-selection-updated (or thread.provider-switched) with the copy, and no command is attached (Orchestrator.ts#L1546-L1549, Orchestrator.ts#L1926-L1941). This event overwrites the newer selection.

The notice had no model of its own. The stored copy was not a user choice, and it was not the model of the run it reports to: on the affected thread, the run that delegated the task was already on the newer model. The copy came from a later turn that ran on the older model.

In #19376, a queued run keeps the model it was sent with, by design. A user message always carries the composer's selection, so it is not affected. A delegated completion queued this way is sent with no model.

Expected: a queued delegated-task completion starts on the thread's current selection and leaves it unchanged, the same as a queued PR-watch update since #21314.

Steps to reproduce

In the app:

  1. In a thread, have the agent start a delegate_task child that takes a minute or more.
  2. Keep the parent thread in a turn when the child finishes, so that the completion notice is queued behind it.
  3. While the notice is queued, change the thread's model, either with t3_thread_configure or the model picker. Both send thread.model-selection.set.
  4. When the running turn ends and the notice starts, the thread's model switches back and the turn runs on the old model.

Deterministic, against the orchestrator on main (05696c6), with the runtimeLayer.test.ts harness:

  1. Create a thread on model A and start a user turn.
  2. Open a delegated-completion cohort on that run. Dispatch the notice the way ProviderContinuationService does: queue_after_active, createdBy: "agent", creationSource: "server", delegatedCompletion, no modelSelection.
  3. Dispatch thread.model-selection.set with model B.
  4. Complete the active run and call resumeQueuedRuns.
  5. The completion run starts on model A, and thread.modelSelection is A again. This also happens when B is on another provider instance.

Version

0.0.46-nightly.20261015.3145 (the local T3 server). Also reproduced on main at 05696c6 with the test above. Triage CLI: 0.1.0.

Environment

Linux x64 (kernel 6.12), Claude Code provider, local T3 server

Evidence

# One thread's orchestration_events on 2026-10-20 (UTC), ids removed.
# A = the older model with effort xhigh; B = the newer model chosen with t3_thread_configure.
11:10:56.299  run.created              run N    queued   command: delegated-completion dispatch  selection A
11:13:18.274  thread.model-selection-updated    command: MCP t3_thread_configure          selection B
11:14:07.399  thread.model-selection-updated    command: none                             selection A
11:14:07.399  run.updated              run N    starting                                  selection A
11:15:47.273  run.created              run N+1  queued   command: delegated-completion dispatch  selection A
11:16:59.896  thread.model-selection-updated    command: MCP t3_thread_configure          selection B
11:17:26.554  thread.model-selection-updated    command: none                             selection A
11:17:26.554  run.updated              run N+1  starting                                  selection A

# Counterexample on the same thread: a configure at 10:34:19 held, because the
# next completion run was created at 10:34:45, after it, and copied B.

Related issues

Fix applied or workaround

Nothing was changed on the machine. Workaround: set the model again after the queued notice starts. A draft PR with a fix and a regression test follows.

Filed by

Claude Code (Claude Opus 5.5) via t3 triage

Activity

  1. added a commit that references this issue on Oct 21, 2026
    6febf86
  2. juliusmarminge commented on Oct 21, 2026

    @juliusmarminge
    Member

    Reproduced on current main 70e5e384da, Linux x64, real web dev server with Claude Opus 5.5. Started an async child, ended the delegating turn, then kept a second parent turn active until the child completion queued. Persisting Sonnet with thread.model-selection.set succeeded at 22:50:09 UTC; queue delivery at 22:50:19 wrote Opus back with no command attached and started the completion on Opus. The picker alone changed the composer draft during the active turn, so the persisted change used the app metadata command. Sanitized runtime transcript.

  3. tim-of-course commented on Oct 30, 2026

    @tim-of-course

    Additional live incident on 2026-10-30: a delegated-task completion switched a thread from Codex / GPT-6.1-Sol back to Claude / Opus 5.5, after the user intentionally left Claude because its usage allowance was exhausted. The automatic Opus turn immediately failed on the usage limit.

    This appears to be the same defect as this issue, with a cross-provider and exhausted-account consequence.

    Observed sequence

    Times are UTC, read from t3_thread_read run metadata and activity (not inferred from the UI clock):

    Event Timestamp Provider / outcome
    Older continuation requested 17:12:26.877 claudeAgent, claude-opus-5-5; subsequently observed queued
    User-selected Sol turn requested 17:14:21.614 codex, gpt-6.1-sol
    Sol turn started 17:14:21.758 Codex
    Sol turn completed 17:21:22.940 Completed successfully
    Older queued continuation started 17:21:23.427 Claude / Opus, about 487 ms after Sol completed
    Continuation completed 17:21:25.008 Failed: Claude usage limit

    At 17:21:23.142, the continuation's activity contains a Codex → Claude provider context handoff followed by the notification that a delegated implementation task had finished. At 17:21:24.963 it records “Claude usage limit reached,” then the provider failure.

    The user explicitly reported: “I switched it to Sol intentionally (I'm out of Opus usage).” No agent-requested provider-switch action appears in the intervening Sol turn. A subsequent user turn ran on Sol again.

    Diagnosis and current source check

    The observed older request starting immediately after the Sol turn, on its previously selected provider, is consistent with the queue-time selection behavior documented above.

    I checked public main at 6fc9144a60c4a5f45089316ddd5f7371bafd3bf2:

    This source check is separate from the incident: the installed server's exact release/commit was not established, and I did not reproduce the incident on current main. The locally available t3 --version reports 0.1.0, but that identifies the CLI, not necessarily the serving process. Environment is Linux with both Claude and Codex providers. No provider calls were made to reproduce the failure, and no runtime configuration or source code was changed during investigation.

    Expected behavior and impact

    An automatic delegate-completion notification should follow the user's current thread selection and should not restore an older provider. If explicit pinning must remain for other queued message types, distinguish those from automatic completions.

    The practical consequence is that switching away from an exhausted subscription can appear to succeed, complete useful work, and then be undone by a delayed background notification. It causes another usage-limit failure and forces the user to restore the intended provider.

    Attaching this evidence to the existing issue rather than opening a duplicate. Private project names, transcript contents, thread IDs and credentials are omitted.

    Reported on the user's behalf by GPT-6.1-Sol.

  4. nhicks00 commented on Oct 31, 2026

    @nhicks00

    Another live incident on 2026-10-31, on 0.1.5 (macOS arm64 desktop app, local server). Here it is two accounts of the same provider (claudeAgent and claudeAgent_2), and the thread loops. The user could not keep it on the account that has quota. This also shows that #22579 alone would not fix it: most of the stuck messages were not delegated completions.

    What the user saw

    The second Claude account (claudeAgent_2) hit its weekly limit (usage_limit_reached, rateLimitType: seven_day, resets Nov 1). The user switched the thread to claudeAgent, which has quota. Each time, the thread switched itself back to claudeAgent_2 1–7 minutes later, and the turn that followed failed on the limit. They switched back five times in about 25 minutes.

    Why it keeps looping

    The thread receives a steady flow of automated messages. While it was busy on claudeAgent_2 overnight, 72 of them queued, each storing claudeAgent_2:

    Kind Count createdBy / creationSource
    Thread-bound webhook scheduled tasks 54 agent / mcp, scheduledTaskId set
    Thread-bound timed scheduled task (heartbeat) 8 agent / mcp, scheduledTaskId set
    t3_thread_send from another agent thread 6 agent / mcp, senderThreadId set
    Delegated-task completions 4 agent / server

    Scheduled tasks take the thread's selection at dispatch (#18467). When the thread is busy, though, dispatch means "queued", so they keep the selection the thread had at that moment. Each cycle went like this:

    1. The user switches to claudeAgent and sends a message. The turn completes.
    2. Per [Bug]: A successful turn on another account releases queued work held by a usage limit on the original account #21705 (and fix(server): held queues release agent and scheduled messages once the thread completes a run #27221), that completed run releases the usage-limit hold even though the queued runs target the exhausted instance.
    3. The next queued run starts with its stored selection (queuedRunModelSelection returns run.modelSelection for non-monitor messages) and emits thread.provider-switched → claudeAgent_2 with no command attached.
    4. It fails on the limit. Any message queued while the thread is back on claudeAgent_2 stores claudeAgent_2 too, so the backlog refills: it grew from 67 to 72 during triage.

    queuedRunModelSelection is unchanged on v0.1.6 and on main. Open PRs #22579 (delegated completions only) and #22425 (provider-specific hold) would together cover 4 of the 72 messages plus the hold release. The 62 scheduled-task messages and 6 agent sends would still switch the thread back as soon as the hold lifts.

    Evidence

    From orchestration_events on the thread (UTC):

    13:21:11.313  thread.provider-switched  cmd: client      -> claudeAgent
    13:28:11.597  thread.provider-switched  cmd: none        -> claudeAgent_2   (queued run starts)
    13:37:36.278  thread.provider-switched  cmd: client      -> claudeAgent
    13:38:26.683  thread.provider-switched  cmd: none        -> claudeAgent_2
    13:41:59.606  thread.provider-switched  cmd: client      -> claudeAgent
    13:43:01.756  thread.provider-switched  cmd: none        -> claudeAgent_2
    13:43:45.979  thread.provider-switched  cmd: client      -> claudeAgent
    13:45:30.042  run 2963 (claudeAgent) completed
    13:45:37.571  thread.provider-switched  cmd: none        -> claudeAgent_2
    13:45:37.571  run.updated  run ordinal 2910 (requested 03:51:10Z, queuePosition 176) starting on claudeAgent_2
    13:45:45.626  run.updated  run 2910 failed  (429 usage_limit_reached, seven_day)
    13:46:49+     new delegated-completion runs created with claudeAgent_2
    

    Workaround that worked

    1. t3_thread_configure the thread onto the working instance first, so new arrivals store it.
    2. Cancel every queued run that stores the exhausted instance (t3_queue cancel, 72 calls).
    3. Re-send their content as one message.

    After that, every queued and running item stored claudeAgent. Disabling the instance does not help: instanceDisabledForProject holds the whole queue, including the runs on the working instance.

    Suggested behavior

    Any queued message a user did not type (createdBy: agent or system, including scheduled-task and agent-sent messages) should start on the thread's current selection, as monitor messages do since #21314. At minimum, a queued run should never switch the thread onto an instance whose usage limit has not reset yet.


    Comment prepared by Claude Code (Claude Opus 5.5) via t3 triage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions