Skip to content

Fork (waits PR 8): slot waits, woken when an agent slot frees - #110

Merged
scuttletoby merged 3 commits into
mainfrom
fork/waitfor-8
Nov 15, 2026
Merged

scuttletoby merged 3 commits into
mainfrom
fork/waitfor-8

Conversation

@scuttletoby

@scuttletoby scuttletoby commented Nov 15, 2026 •

Copy link
Copy Markdown
Owner

Waits part 8 (slot waits). Off by default, like every waits part: with waits off nothing runs, nothing is read or written, and no held slot counts (tested).

What it adds

  • Slot waits. wait_for {"kind":"slot"} (optionally {"slot":{"account":...}}) is met when the slot gate (feat(slots): agent slot gate on delegate_task (off by default) #89) has a free agent slot for the thread. Built on the gate and on the waits worker (feat(waits): agent tools to register, list and cancel waits (part 4, off by default) #103).
    • Free at registration: met at once, nothing stored. The slot is held for the thread for agentSlots.reserveSeconds (default 120). The answer says "call delegate_task now".
    • Full: stored as waiting. Each sweep asks the gate for slots, oldest first (created time, then id). Each granted slot is held for its thread at once, so one free slot wakes one wait, never two. Grants in one sweep count against the free slots even with reserveSeconds: 0, so wakes per sweep never exceed the free slots.
    • No jumping the queue: a new slot wait behind older waiting slot waits is stored and waits for the sweep, even if it sees a free slot.
  • Held slots (spec 4.6). A held slot counts as a running agent for every other thread, on the machine and on its account when the wait named one. The holder's next allowed delegate_task uses it up.
    • At most one held slot per thread. A repeat ask (met at register or granted in a sweep) replaces the thread's earlier hold, it never adds one. Counts and wait_list count one hold per thread. A second slot wait of a thread granted in the same sweep asks again next time.
    • A parent and its children may each hold one. That stays under the limit: each hold counts toward the machine and account limits, and each child that asks is itself a counted running agent.
    • The gate keeps grants in memory. A delivered slot wait keeps reservedUntil in the store, so a held slot survives a restart.
    • Using the slot clears reservedUntil. If that write fails, the stored copy still stops counting and ends by itself.
    • A held slot never lets anyone past a limit: the holder's own delegation is counted like any other.
    • Held slots count only while waits are on. The store is read only then.
    • Only a granted wait holds a slot. Timed out, handed over to a parent, slots off or reserveSeconds: 0 holds nothing. A granted wait handed over (or ended without a receiver) before its wake is sent frees its slot at once.
    • A stored reservedUntil is capped at endedAt (or now) + reserveSeconds on load and on read, so a clock that was ahead or a hand-edited file cannot block others for longer, and the record is pruned as usual.
  • Counts that cannot be read (projection error, timeout, a crash in the read) are a failed read: error grace, then check_error. Never "free", and nothing is held.
  • Snapshot read only when needed (check note 3.6): only while slots are on and a count limit applies to the request. A wait with no account under a memory-floor-only limit reads only memory.
  • Slots off or limits changed while waiting: the next sweep uses the new settings. A raised limit wakes waits. Slots turned off wakes every slot wait with "Agent slots are off, so delegate_task is not held back." and holds nothing.
  • Refusal text: while waits are on, a refused delegate_task names wait_for {"kind":"slot",...} (from feat(slots): agent slot gate on delegate_task (off by default) #89; the wording comes from slotRefusalReason).
  • wait_list shows slot usage (machine and per account: running + starting + held) only while slots are on.
  • One gate: WaitWorker gets the server's one SlotGate (the same layer, which is memoized by reference), so slot waits and delegate_task share its semaphore and memory.

Shared registry/judge code touched (kept small for PR 6's rebase): register picks the gate instead of a judge for kind: "slot" (one ternary), judgeSweep skips slot waits, and runDue gets one "grant" phase after "judge".

The state question

What happens when this runs again, stops half-way, or meets existing/old/empty/corrupt/newer-version state, or a server restart?

Stored or written thing Case What happens Test
Slot wait record (waiting) Slot frees while the wait is being made It is stored as waiting and is due at once. The next sweep wakes it. a slot that frees while the wait is being made: ...
Slot wait record Two waits, one free slot; sweep runs twice Only the oldest is woken. Its slot is held, so the second sweep wakes nobody. A second delegate_task by the holder is refused. two waits for one free slot: ... (service and gate)
Slot wait record Slots turned off / limit raised while waiting Raised limit: woken at the next sweep. Off: woken with "slots are off", nothing held. slots turned off while waiting: ..., the limit raised while held: ...
Slot wait record Waiting thread archived Handed to the parent (existing hand-over). It is never granted and holds no slot. an archived waiter: ..., shared holds a slot only for a granted slot wait
Slot wait record Waiting thread stopped Granted, message held (existing stop rules). The gate's held slot ends after reserveSeconds, so a stopped thread does not block others. a stopped waiter: ...
reservedUntil on a delivered slot wait Server restart The new gate counts it from the store. The holder's delegate_task uses it and clears it. Open waits are granted by the new server. server restart: ... (service), slots held in the wait store: ... (gate)
reservedUntil Clearing write fails (stops half-way) The acquire still allows. The stored copy stops counting in this server and ends by itself at reservedUntil. a used slot whose store write fails ...
reservedUntil Old store (written before this PR) The field was already optional in the contract (#90). Old files load unchanged, and nothing is held. existing store tests
Grant write (slotGranted) Write fails, or the wait was cancelled meanwhile The grant is dropped from the gate (holds nothing) and is tried again at the next sweep. a held slot ends after reserveSeconds, and a dropped grant holds nothing
Grant Waits turned off mid-sweep Nothing is written, and the grants are dropped. covered by the stillOn path (same as the judge sweep)
Gate memory (held slots) Server restart Lost (memory only). A wait met at registration loses its hold. Accepted: it costs one more wait. Which stored holds were used or replaced is memory too: after a restart such a stored hold counts until it ends (the safe side), at most one per thread. server restart: ..., two stored held slots of one thread (a restart): ...
Held slots Same thread asks again (runs again), many met slot waits, or a sweep grants two waits of one thread One hold per thread: the new ask replaces the old one (memory and stored). In one sweep the second wait asks next time. A refused ask keeps the hold it has. one thread holds at most one slot: ..., a woken thread that asks again keeps one slot: ..., gate one held slot per thread: ..., two slot waits of one thread in one sweep: ..., a thread already holding a slot that is refused ..., shared count at most one per thread: ...
Grants in a sweep reserveSeconds: 0, one free slot, several waiters One wake per free slot per sweep; the rest wait. reserveSeconds 0: one free slot wakes one waiter per sweep ..., gate reserveSeconds 0: one free slot wakes one wait ...
reservedUntil Far in the future (clock was ahead, hand-edited or corrupt value) Capped on load and on read at endedAt/now + reserveSeconds; after that it is expired and pruned with the record. a stored held slot far ahead ...
Granted wait Thread archived after the grant, before the wake is sent (stops half-way) Handed to the parent, and the gate's hold is dropped at once. a granted wait handed over to the parent frees its slot at once
Granted wait Holder delegates before its wake is delivered The delivery records the slot as used, so reservedUntil is not written again. a slot used before its wake is delivered is not held again by the delivery
Slot snapshot Cannot be read unread → failed read → error grace → check_error; never "free". It clears when the read works again. counts that cannot be read: ..., running agents that cannot be counted: ...
Store Corrupt or newer version Reported as before. The gate counts no stored held slots from it. existing store tests + heldInStore returns none when not loaded
Everything Waits off (default) The store is never read or written, and stored held slots do not count. Slots off as well: no snapshot read. waits off (the default): ..., off, or no limit: ...

Red-check: every new test (26) fails on origin/main sources and passes with this change. The hand-over rule was also red-checked alone. Fix round: 12 new tests; 11 fail on the previous head (9804190) and pass now. The 12th (a thread already holding a slot that is refused keeps the slot it holds) guards the new replace path and passes on both by design.

Tests: 278 passed across the waits, slots and shared slot/wait test files (fix round). Typecheck (server, shared), lint, format and knip are clean.

Accepted limits (by design, noted for review):

  • A direct delegate_task can take a freed slot before the next sweep grants it to a waiting wait. The waiting wait waits for the next free slot.
  • A holder archived or stopped after its wake was delivered keeps its hold until reserveSeconds ends. The gate reads only threads with an active run, so it cannot cheaply tell an archived holder from an idle one.
  • A slot used before its wake is delivered: the wake message still says the slot is held (the hold itself is not written).
  • A thread can renew its one hold by asking again. It never holds two.
  • A delegate_task dispatch that fails after the gate allowed it has used up the hold.

General vs project

All general. Limits, reserveSeconds and waits on/off are T3 settings (Settings UI, #104). Accounts are provider instance ids. Nothing project-specific is in the code.

Not in this PR

#95 and #105 stay open. PR 6 (pr judge) is built in parallel; this PR touches only the slot parts of the shared registry code.

Spec: the F3 waitfor spec (fable-2026-11-13), section 4.6. Scope and decisions: check note waitfor-check-2026-11-14, sections 3.1, 3.6 and the PR 8 verdict in section 6. Where the two differ, the check note wins.

🤖 Generated with Claude Code

A slot wait (wait_for kind "slot") asks the slot gate for a free agent slot.
Free at once: met, and the slot is held for the thread for reserveSeconds.
Otherwise the sweep grants slots oldest first, holds each one for its
thread, and wakes it; the thread's next allowed delegate_task uses the slot
up. Held slots count as running agents for every other thread, only while
waits are on. Counts that cannot be read are an error, never "free". With
slots off, a slot wait is met at once and holds nothing. Waits off: nothing
of the store is read or written, and no held slot counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scuttletoby

Copy link
Copy Markdown
Owner Author

lens: concurrency-and-abuse · reviewer: opus high (t3lens110a) · read at: 9804190

covered: git diff a6a87b3700...9804190b07 (SlotGate.ts, WaitService.ts, WaitWorker.ts, runtimeLayer.ts, shared agentSlots.ts / waitFor.ts and their tests), plus the code it calls: acquire/decide, the register path, grantSweep/applyGrants, the deliver sweep's hand-over, the due hint, and ensureLoaded/commitWaits. Also open PR #109 (PR 6) against this head.

checks:

  • Races. Two registrations for one free slot are run one at a time by the gate semaphore; the second sees the first's in-memory hold and is stored as waiting (the existing test passes). Registering during a sweep: the queued rule plus the shared semaphore mean the worst case is a newer wait being served first (fairness only, never a double start). delegate_task while a grant lands: grant and decide share one semaphore, and a grant is held in memory before anything is written, so a delegation can never take it. Lock order is always gate semaphore → store lock (heldInStore). Nothing takes the semaphore while holding lock (drop and usage don't), so it cannot deadlock. Server stop part-way through a grant: the hold is in memory first, delivering is written before the send, and reservedUntil is written at delivered. After a restart the message id replays (no second wake), but the slot is not held between the restart and the re-delivery, so the "slot is held" text can be wrong once. Low; it fixes itself.
  • Can a held slot push a thread over the limit? No. A hold only lowers what other threads see. The holder's own holds are left out of its count, but it still needs running + starting + other threads' holds < max. Using a hold swaps it for a starting entry under the same semaphore, and usedKeys stops the stored copy from counting twice. grant keeps PR 7's rule that the caller's own run is not counted, so nothing new can overshoot.
  • Parent and child. A child cannot use its parent's hold and a parent cannot use its child's: the exclusion matches the exact thread id, and a handed-over wait never gets reservedUntil.
  • Wake messages. They go only to the wait's thread, or to its parent on hand-over (no hold then). The text gives the count and the provider instance id only. Note: while slots are on, wait_list shows every thread the machine and account counts of all projects. Low and fine for one owner.
  • PR 6 (feat(waits): the pr judge (part 6, off by default) #109). git merge-tree against feat(waits): the pr judge (part 6, off by default) #109's head is clean, and the two do not conflict in meaning: feat(waits): the pr judge (part 6, off by default) #109 adds the pr judge and changes STOPPABLE_POOLS; slot waits bypass judges and are left out of the judge sweep's due list. The only effect: the grant phase runs after the judge phase, so slow pr judges delay grants by up to the judge window (speed only).
  • Gate wiring. I checked with a small layer test that a layer given inside a worker with Layer.provide and also merged at the top is built once. So the wait worker and /mcp share one gate, as the comment in runtimeLayer.ts says.
  • State question.
    • Runs again: a stored waiting wait is deduped by key, but a met slot wait is never stored, so nothing dedupes or caps it (finding 1).
    • Stops half-way: a grant whose write fails, or whose wait was cancelled, is dropped (drop); waits switched off between grant and write → dropped and nothing written. Good.
    • Old, empty, corrupt or newer-version store: heldInStore returns [], so stored holds stop counting (memory holds still count). That under-counts for at most reserveSeconds; acceptable.
    • Restart: memory holds, starting entries and usedKeys are lost; stored reservedUntil still counts. A used hold whose clearing write failed counts again until it ends; that over-counts, which is the safe side.
  • Tests run (scratch worktree at the head, no build): WaitService.slots.test.ts 12/12 pass; the 2 lens tests below fail on the head; the layer-memo test passes.

finding: MEDIUM (abuse: one thread can take every free slot) apps/server/src/waits/WaitService.ts:634, apps/server/src/slots/SlotGate.ts:322, apps/server/src/slots/SlotGate.ts:220

  • Cause: a slot wait that is met at registration holds a slot in gate memory under a fresh wait id. Nothing is stored, so maxPerThread and the key dedupe never apply. Each repeat wait_for {"kind":"slot"} from the same thread takes another free slot for reserveSeconds (default 120). grant counts the thread's earlier holds only to stop two holds being taken for one free slot.
  • Scenario: max 3, nothing running. T1 calls wait_for slot 3 times and gets met 3 times, starting nothing. Every other thread's delegate_task gets slots_full, and their slot waits stay full. T1's own delegations still pass, because decide leaves all of the caller's holds out of its own count. Calling again after each expiry keeps every slot held indefinitely. There is no per-thread or per-project cap on holds. It does not need malice: an agent that "checks" for a slot before each step, or fans out wait_for calls in parallel and then does fewer delegations, does this.
  • Fix: at most one live hold per thread (per account). When registering or granting for a thread that already holds a live hold covering the request (memory or store), answer met and extend that hold instead of adding one.
  • Failing test: one thread holds at most one slot: repeated met slot waits do not hoard every free slot (expects OUTSIDER allow; the head gives refuse).

finding: LOW (duplicate wakes, against R17) apps/server/src/slots/SlotGate.ts:322

  • Cause: with agentSlots.reserveSeconds: 0 (allowed by the contract), grant does not add a grant to the sweep's local reservations. So one free slot grants every waiting slot wait in that sweep.
  • Scenario: max 1, 3 waiters, the one agent ends: all 3 threads are woken ("Call delegate_task now"). That is 3 paid turns; 1 gets the slot and 2 get slots_full and register again. Spec 4.6 says grants made earlier in the sweep are counted for the later ones, so a sweep wakes as many waits as there are free slots.
  • Fix: always push the grant into the local reservations array; skip only the held Ref (and so the stored hold) when the reserve is 0.
  • Failing test: reserveSeconds 0: one free slot wakes one waiter per sweep, not all of them (R17) (expects [thread-1]; the head gives [thread-3, thread-1, thread-2]).

notes (low, end on their own within reserveSeconds, no test):

  • (a) If the holder delegates between grant and delivered, usedKeys ends at the memory hold's end, while the store's reservedUntil (set later, at delivery) then counts again for the length of the delivery delay. The woken message also tells the thread about a hold it has already used.
  • (b) A wait granted and then handed over to the parent (receiver archived between grant and delivery) keeps its memory hold, keyed to the gone thread, for reserveSeconds.
  • Either could be closed by calling gate.drop([id]) when delivery ends with handedOverBecause, or when the hold was already used.

verdict: NOT SAFE. Fix finding 1, a small change: one hold per thread. Finding 2 is a one-line change. With those fixed, everything else on this lens holds.

@scuttletoby

Copy link
Copy Markdown
Owner Author

lens: correctness · reviewer: opus high (t3lens110c) · read at: 9804190

covered: git diff main...HEAD (SlotGate.ts, WaitService.ts slot parts, WaitWorker/runtimeLayer wiring, shared agentSlots.ts + waitFor.ts slotUsed/delivered), plus the code it calls: OrchestratorMcpService slot hook, handOver/threadStopped, applyJudgements/nextDueAt/runDue, ensureLoaded, the lock order between the gate semaphore and the wait-store lock. Also checked against PR 6 (#109 at 61cb512).

checks:

  • (1) Release: every held slot ends by itself. Memory holds end at grant time + reserveSeconds. Stored ones end at reservedUntil, which is in the past after a restart and so filtered out. The holder's allowed delegate_task uses up its hold (memory removed, key in usedKeys, store cleared outside the semaphore; a failed clear stops counting here). Cancelled between grant and write: applyGrants drops it. Waits off: no hold counts (liveHeld returns []). Slots off or no limit: grant returns off and reserveSeconds is treated as 0. A limit change is read on the next call. A stopped, archived or deleted holder is not released at once, but its hold ends within reserveSeconds. Exception: a stored reservedUntil far ahead (finding 2).
  • (2) Order and no double grant: grant and acquire share one semaphore; each grant is added to the counted holds before the next request. Waits are sorted by createdAt, then id. A grant made at registration and one made by the sweep are serialised by that semaphore, and a slot wait met at registration is never stored, so the same key is never granted twice. Two sweeps: the first one's hold is counted by the second (tested in the PR). Exception: reserveSeconds: 0 (finding 1). Lock order is always gate semaphore -> store lock; applyGrants calls only drop, which takes no semaphore, so no deadlock.
  • (3) Unreadable count: grant returns unread, which becomes a failed (not permanent) check, then error grace, then check_error. At registration it is stored as waiting, never met. An unreadable store gives no waiting waits and no holds.
  • (4) The two added rules are right. A hand-over turns a waiting wait into delivering with no result, so it is never granted. A granted wait that is handed over gets no reservedUntil. Slots off or no limit: stateSettings.reserveSeconds = 0, and the gate holds nothing.
  • (5) Known limits are acceptable. First: a direct delegate_task taking a freed slot before the sweep keeps the limit and only affects fairness (R17 is about fairness among slot waits). Second: a hold kept only in memory and lost on restart is either a registration grant (costs one more wait) or a grant not yet delivered (delivery after the restart writes reservedUntil again). Neither leaks a slot or goes over the limit.
  • (6) Waits off: register refuses, runDue returns early, liveHeld neither reads the store nor counts holds, and useStoredSlot checks enabled. Slots off: decide, grant and usage return before readRunning. In the sweep the snapshot is read only when there is a waiting slot wait and a count limit applies (3.6).
  • One gate: confirmed that the same layer value used in a nested Layer.provide and in an outer provideMerge is built once (scratch test, passes). So WaitWorker and /mcp share one SlotGate.
  • PR 6 (feat(waits): the pr judge (part 6, off by default) #109): git merge-tree is clean, and the meaning does not conflict. feat(waits): the pr judge (part 6, off by default) #109 adds a pr judge and narrows STOPPABLE_POOLS, and pr waits go through judgeSweep, which this PR changes only to skip slot waits.
  • State question. Runs again: a second sweep or a second slotUsed writes nothing. Stops half-way: a failed grant write drops the memory grant and the next sweep tries again; a failed clear means the stored hold ends by itself. Old store: reservedUntil was already optional. Corrupt or newer store: unavailable, so no holds and no waits. Restart: covered in (5). Empty store: nothing to do. Clock jump or bad timestamp: finding 2.
  • Tests: SlotGate, SlotGate.lens110, WaitService.slots and agentSlots tests run at head: 62 PR tests pass; 2 lens tests fail as expected (the findings). CI green.
  • Notes (not findings): (a) if a dispatch fails after acquire, the holder's hold is already used and lost; release gives back only the starting entry. (b) A granted wait whose thread is archived before delivery reaches the parent as "…slot: free. Call delegate_task within N s; the slot is held for this thread." The gate still counts the child's memory hold against the parent for up to reserveSeconds, so the parent can be refused. Possible fix: drop([id]) on hand-over, and reword the message. (c) With only a memory floor, every waiting slot wait is woken at once. That is consistent with the floor in PR 7, which is not a count.

finding: minor · apps/server/src/slots/SlotGate.ts:322 · With agentSlots.reserveSeconds: 0 (an allowed setting, nonNegativeIntDefault), grant adds no reservation, so one sweep grants every waiting slot wait for a single free slot. Scenario: maxPerMachine: 1, nothing running, 3 waiting slot waits: the result is ["granted","granted","granted"], so 3 threads are woken with "free" and 2 are then refused with slots_full. This breaks R17 ("as many wakes as free slots"). Fix: push the grant into the local reservations for the rest of this pass even when reserveMs is 0; only skip the held Ref. Failing test: reserveSeconds 0: one free slot wakes one wait, not every wait.

finding: minor · apps/server/src/waits/WaitService.ts:265 (and SlotGate.ts:179) · A stored reservedUntil is used without any upper bound. If the clock was ahead when it was written, or the store was hand-edited or corrupt (e.g. a year-9999 date, which the schema accepts), every other thread is refused until that time. shouldPruneWait also keeps the record while reservedUntil is in the future, so it is never pruned. Only the holder's own delegate_task clears it, and an archived holder never sends one. Scenario: a stored hold 30 days ahead, reserveSeconds: 60, maxPerMachine: 1: a different thread is still refused 2 minutes later. Fix: cap the stored hold at min(reservedUntil, now + reserveSeconds) when reading it (and the same in nextDueAt/prune). Failing test: a stored held slot never counts longer than reserveSeconds from now.

findings: 2 (both minor; no blocking issue)
verdict: SAFE (fixing both before ready is cheap and recommended)

…d stored holds

- A thread holds at most one slot. A repeat ask (met at register or granted in
  a sweep) replaces its earlier hold instead of adding one; counts and the
  usage view count one hold per thread (onePerThread).
- Grants in one sweep count against the free slots even with reserveSeconds 0,
  so one free slot wakes one waiter; a second wait of a granted thread asks
  again next time.
- A stored reservedUntil is capped at endedAt/now + reserveSeconds on load and
  on read (clock ahead or hand-edited file), so an expired record is pruned.
- A granted wait handed over to the parent, or ended without a receiver, frees
  its slot at once; a slot used before its wake is delivered is not held again
  by the delivery.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scuttletoby

Copy link
Copy Markdown
Owner Author

not a lens: fix note

Fix round on 997e16a (one commit: "Slot waits: one held slot per thread, wakes within free slots, bounded stored holds"). Each finding -> commit + test:

  1. MEDIUM, hoarding (concurrency and abuse review): one thread could hold every free slot through repeated met slot waits. -> 997e16a. A thread now holds at most one slot. A repeat ask, whether met at register or granted in a sweep, replaces the earlier hold (memory and stored), and counts use one hold per thread.
    • WaitService.slots.test.ts: one thread holds at most one slot: repeated met slot waits do not hoard every free slot, a woken thread that asks again keeps one slot: the stored one is replaced
    • SlotGate.test.ts: one held slot per thread: asking again moves and renews it, never adds one, two slot waits of one thread in one sweep: one wake, the other asks next time, two stored held slots of one thread (a restart): counted once, both replaced by a new ask, a thread already holding a slot that is refused keeps the slot it holds
    • agentSlots.test.ts: count at most one per thread: the latest end, then the last key
    • A parent and its children each hold at most one. That is fine under the limit: every hold counts toward the machine and account limits, and a child that asks is itself a counted running agent.
  2. LOW, reserveSeconds 0 wakes everyone (both reviews) -> 997e16a. Grants in one sweep count against the free slots even when nothing is held.
    • WaitService.slots.test.ts: reserveSeconds 0: one free slot wakes one waiter per sweep, not all of them (R17)
    • SlotGate.test.ts: reserveSeconds 0: one free slot wakes one wait, not every wait
  3. LOW, unbounded stored reservedUntil -> 997e16a. Capped on load and on read at endedAt (or now) + reserveSeconds, so the record expires and is pruned as usual.
    • WaitService.slots.test.ts: a stored held slot far ahead (clock ahead, hand-edited file) ends reserveSeconds after it is read, and is pruned
    • The review's gate-level case with a mocked store became this service-level test, because the cap lives where the store is read.
  4. Notes:
    • A used hold briefly counts again -> 997e16a, fixed. A slot used before its wake is delivered is recorded as used, so the delivery does not write reservedUntil again. Test: a slot used before its wake is delivered is not held again by the delivery.
    • A granted wait handed over before its wake is sent -> 997e16a, fixed. Its hold is freed at once. Test: a granted wait handed over to the parent frees its slot at once.
    • An archived or stopped holder after delivery: not changed. It keeps its hold until reserveSeconds ends (default 120 s, bounded by fix 3). The gate reads only threads with an active run, so it cannot cheaply tell an archived holder from an idle one. Listed in the PR body's known limits.
    • A dispatch that fails after the gate allowed it loses the hold: not changed. The retry asks the gate again like any delegation. Listed in the known limits.
    • Restart: which holds were used or replaced lives in memory. After a restart, such a stored hold counts until it ends: the safe side, and at most one per thread.

Red-check: 12 new tests. 11 fail on the previous head 9804190 and pass now. The 12th (a thread already holding a slot that is refused keeps the slot it holds) guards the new replace path and passes on both by design.

Checks: 278 tests pass across the waits, slots and shared slot/wait files. Server and shared typecheck, lint, format and knip are clean. The PR body's state rows and known limits are updated.

🤖 Generated with Claude Code

@scuttletoby

Copy link
Copy Markdown
Owner Author

lens: correctness · reviewer: opus high (t3lens110q) · read at: 997e16a

covered: git diff 9804190b07..997e16aba6 (SlotGate.ts useUp/clearStored/grantLocked/grant/decide/usage, shared onePerThread and boundHeldSlot, WaitService storedReservation/heldInStore/useStoredSlot/usedBeforeDelivery/applyGrants/the delivery hand-over drop/ensureLoaded/nextDueAt), plus the code it calls: slotAtRegister and its queued rule, grantSweep order, the delivered and slotUsed transitions, shouldPruneWait. Read against both earlier lenses and the fix note. Also the PR merged with current main (PR 6 merged as 09f1439).

checks:

  • (1) One hold per thread. I checked every path that adds a hold:
    • register met, sweep grant, and a re-ask: useUp moves the thread's memory and stored holds before the new one is added. grantedThreads stops a second grant in one call (sameThread).
    • hand-over: a granted, handed-over wait is dropped at once and never gets reservedUntil.
    • used before delivery: usedBeforeDelivery stops the delivery from writing reservedUntil again.
    • restart: memory is gone. Stored copies whose clear failed are counted once by onePerThread and are all replaced by the next ask.
    • Result: I found no path that leaves two live holds for one thread.
  • Replace never frees a slot another thread already got. The holder is granted only when it fits with its own hold left out. Every other thread counted that hold, so no one else can have taken it. A refused re-ask keeps the old hold (tested).
  • Parent and children. Each one's hold counts for everyone else, so the holds plus the running agents stay within the limit when each grant is made. The caller's own run is still left out, the same rule as in PR 7. Nothing new can overshoot.
  • (2) Wakes per sweep ≤ free slots, including with reserveSeconds 0: the local reservations now always takes the grant. The SlotGate and WaitService tests pass. A holder's own re-ask in the same sweep renews its own slot and takes no new one.
  • (3) Bounds.
    • boundHeldSlot caps a stored hold at load to min(now, endedAt) + reserveSeconds, or removes it. storedReservation caps it again on each read to endedAt + reserveSeconds with the current setting, so a lower setting takes effect at once.
    • delivered always writes endedAt, so a record with no endedAt (old or damaged) holds nothing. That is the safe side.
    • Pruning reads the capped reservedUntil, so a year-9999 record is pruned as usual (tested).
    • The gate-level lens test with a mocked store still fails, by design: the cap is where the store is read. I accept the service-level test that replaced it.
  • (4) The two fixed notes are correct (used-before-delivery; handed-over grant dropped). The two known limits left are acceptable:
    • An archived holder keeps its hold for at most reserveSeconds (bounded now).
    • A failed dispatch uses up the hold. The retry simply asks again.
    • New, same kind of limit: if a holder's re-ask is cancelled, or its write fails between the grant and applyGrants, the old hold was already moved, so the thread loses its slot. That is the safe side. Follow-up note only.
  • (5) Round 1 still holds.
    • No double grant: one semaphore, and the grant is held before it is written.
    • An unreadable count is unread and never "free".
    • Waits off: liveHeld reads nothing. The new readSettingsOrNull calls in heldInStore and ensureLoaded read settings only, never the store.
    • Slots off: grant returns off before any snapshot read.
    • Lock order is still gate semaphore → store lock. clearStored runs outside the semaphore.
  • (6) PR 6 on main.
    • git merge-tree origin/main 997e16aba6 is clean. The meaning does not change: PR 6 adds the pr judge, and slot waits skip judges.
    • The scratch merge passes SlotGate*, WaitService.slots* and the other waits tests, plus shared agentSlots and waitFor (97/97).
    • One test fails on the merge: WaitPrJudge > a head is settled only by two reads of its own checks. It fails the same way on main alone in my sandbox, while main's CI run passes it. So this PR does not cause it, and it is likely something in my local setup. A follow-up should check it.
  • State question: "What happens when this runs again, stops half-way, or meets existing/old/empty/corrupt/newer-version state, or a server restart?"
    • Runs again: a re-ask moves the hold and a second sweep adds nothing. But a holder that re-asks again and again keeps its slot forever (finding).
    • Stops half-way: a grant that is not written is dropped. The moved old hold is lost, which is the safe side.
    • Existing or old state: a stored hold with no endedAt holds nothing. Pre-fix duplicates are counted once.
    • Empty, corrupt or newer-version store: no holds and no waits.
    • Restart: stored holds are capped on load. A hold replaced before the restart whose clear failed counts until it ends, at most one per thread.
  • Tests run (scratch worktree, no build):
    • At the head: waits + slots, 186 pass, 2 fail. One is the replaced gate-level lens test (by design). The other is my new test below.
    • On the merge: 201 pass, 3 fail: those two plus the WaitPrJudge test that also fails on main here.
    • All 12 fix tests pass.

finding: MEDIUM (fairness / starvation, new in this round) · apps/server/src/slots/SlotGate.ts:337 (own hold left out) and apps/server/src/slots/SlotGate.ts:349/:357 (new end = now + reserveSeconds, old hold moved)

  • Cause: a thread that already holds a slot is granted again with its own hold left out of the count, and the hold's end is renewed to now + reserveSeconds. The queued rule sends the re-ask to the sweep when others wait, and the sweep goes oldest first. But the older waiter of another thread is refused, because the holder's slot counts against it. The holder's newer re-ask is then granted, because its own slot does not count. So the newest asker beats the oldest waiter, every time.
  • Scenario: maxPerMachine: 1, reserveSeconds: 60. T1 is granted the slot. T2 waits. T1 calls wait_for {"kind":"slot"} again every ≤ 60 s and never starts an agent: each sweep gives T2 full and T1 granted (renewed). T2 starves until its own timeout, and every other delegate_task gets slots_full.
  • Round 1 counted the thread's own holds in grant, so T1's re-ask was refused and T2 got the slot when T1's hold ended. This regression comes from the hoarding fix.
  • The trigger is the one the abuse lens called realistic: an agent that checks for a slot before each step, or re-asks after it is woken instead of delegating.
  • Fix (small): moving a hold must not extend it. When the thread already holds a slot, use untilMs = min(its current end, now + reserveSeconds). A narrower option: renew only when no other thread's slot wait was refused earlier in this call. After that, the end of the first hold hands the slot to the oldest waiter.
  • Failing test (gate level, kept by the reviewer): a holder that keeps asking again never keeps its slot past reserveSeconds while an older wait of another thread waits. It expects T2 to be granted within 10 rounds of 50 s. The head gives full every time.

follow-ups (minor, no new round): (a) a cancelled or unwritten re-ask loses the hold it moved (safe side; could keep the old hold until applyGrants keeps the new one). (b) check why WaitPrJudge > a head is settled only by two reads of its own checks fails locally on main while CI passes.

findings: 1 (medium)
verdict: NOT SAFE. One small change: a moved hold keeps its earlier end. Everything else in this round is correct: one hold per thread, wakes within free slots, bounded stored holds, both notes fixed, and nothing from round 1 broken.

A thread that kept asking without starting an agent renewed its hold on every ask, so an
older wait of another thread never got the slot. The moved hold now keeps the earlier end.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scuttletoby

Copy link
Copy Markdown
Owner Author

not a lens: fix note

Fix for the medium finding of the final check (#issuecomment-6887477406), commit 9209261: when a thread asks again, its hold moves to the new ask but keeps the earlier end time, so a holder that keeps asking without starting an agent loses the slot at the first end and an older waiter of another thread gets it.

🤖 Generated with Claude Code

@scuttletoby
scuttletoby marked this pull request as ready for review November 15, 2026 01:47
@scuttletoby
scuttletoby merged commit 815eeaa into main Nov 15, 2026
25 checks passed
@scuttletoby
scuttletoby deleted the fork/waitfor-8 branch November 15, 2026 01:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant