Repository navigation
fix(macos): serialize heartbeats through a retrying queue, create the bucket lazily - #162
Conversation
… bucket lazily Every poll and focus change spawned its own Task: when the server was slow they piled up in parallel (six in flight observed during an aw-server stall) and a failed heartbeat was dropped, so the time until the next successful send was credited to whatever window was sent next. Bucket creation was fire-and-forget too, so a watcher started before the server (login) never got a bucket and every heartbeat 404ed. HeartbeatSender is an actor with a bounded in-memory queue: one request in flight, in order, failed heartbeats retried with exponential backoff (1 s to 60 s), pulsetimes computed at send time relative to the last accepted heartbeat so the server-side merge still covers the gap, and the bucket created (or confirmed) before the first send with the same retry. Verified with a harness built from this file's sender code against an aw-server-rust started 5 s after the watcher: 24 heartbeats over 12 s, alternating windows every 3 s, retried at 1/2/4 s, all delivered in order and merged into 4 events of 3 s (last 2.5 s). Fixes #160.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 28094d04fe
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let heartbeatSender = HeartbeatSender() | ||
|
|
||
| func sendHeartbeat(_ heartbeat: Heartbeat) { | ||
| Task { await heartbeatSender.enqueue(heartbeat) } |
There was a problem hiding this comment.
Preserve producer order before enqueueing
Each call launches an independent unstructured Task, and Swift does not guarantee that these tasks submit their actor calls in creation order. With back-to-back window callbacks, a newer heartbeat can therefore reach enqueue first; when the older one arrives, its negative interval relative to lastEnqueued satisfies <= 0.5 and it is discarded. Feed the actor through a single ordered producer or another synchronously ordered mechanism so the new sender actually preserves callback chronology.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 2404d56: the sender is now a lock-protected class and enqueue runs synchronously on the calling (main) thread, so call order is delivery order; only the drain runs in a Task.
| if queue.count >= Self.maxQueued { | ||
| queue.removeFirst() | ||
| droppedSinceLastLog += 1 |
There was a problem hiding this comment.
Avoid removing the in-flight heartbeat from the bounded queue
When the queue is full while drain is suspended in createBucket or send, actor reentrancy allows this removal to delete the head already captured as next. If that request succeeds, drain subsequently calls queue.removeFirst() again and deletes the following heartbeat without sending it. This is especially plausible when recovering with a full backlog; keep the in-flight item separate from the pending queue or remove it by verified identity after success.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 2404d56: the heartbeat being sent is taken out of the queue into a separate in-flight slot and only cleared on success; dropping the oldest pending entry can no longer touch it. Harness re-run gives the same 4 merged events / 11.6 s.
|
| bucketReady = true | ||
| } | ||
| try await send(next) | ||
| queue.removeFirst() |
There was a problem hiding this comment.
Recovery drops an unsent heartbeat
When the queue is full, enqueue can remove the heartbeat that drain is currently sending while send(next) waits for the server. After that send succeeds, this removeFirst() removes a second heartbeat that was never sent. Recovery after a long outage can therefore lose window changes beyond the intended oldest-entry drops.
Keep the in-flight heartbeat separate from the queue, or remove only the entry that actually finished sending.
There was a problem hiding this comment.
Already addressed in 2404d56 (pushed just before this review): enqueue is synchronous on the calling thread under a lock, so callback order is delivery order, and the heartbeat being sent lives in a separate in-flight slot that a full-queue drop cannot touch.
| func sendHeartbeat(_ heartbeat: Heartbeat) { | ||
| Task { await heartbeatSender.enqueue(heartbeat) } | ||
| } |
There was a problem hiding this comment.
sendHeartbeat still launches an independent Task for each callback. The actor receives heartbeats in task scheduling order, not necessarily callback order. If a later heartbeat reaches enqueue first, the earlier heartbeat has a negative time difference and is discarded by the <= 0.5 check—even when the observations were more than half a second apart. That loses a window change and leaves the promised ordering fix incomplete.
Put heartbeats into one ordered stream before the asynchronous handoff.
There was a problem hiding this comment.
Already addressed in 2404d56 (pushed just before this review): enqueue is synchronous on the calling thread under a lock, so callback order is delivery order, and the heartbeat being sent lives in a separate in-flight slot that a full-queue drop cannot touch.
There was a problem hiding this comment.
Thanks for the update. I rechecked 2404d56: sendHeartbeat now calls heartbeatSender.enqueue synchronously, and enqueue serializes queue mutation under NSLock, so callbacks on the main thread preserve delivery order before the asynchronous drain begins. The separate inFlight slot also prevents a full pending queue from dropping the heartbeat currently being retried. This resolves my ordering and in-flight queue concerns.
…t of the bounded queue Review follow-ups on #162: reaching the actor through a detached Task per call did not guarantee that back-to-back callbacks arrived in order, and an older heartbeat arriving second was dropped by the sub-second dedupe; and dropping the oldest pending entry while a send was suspended could remove the very heartbeat being sent. The sender is now a lock-protected class whose enqueue runs synchronously on the caller (main thread), and the heartbeat being retried is held apart from the pending queue.
|
@greptileai review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2404d5638a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if queue.count >= Self.maxQueued { | ||
| queue.removeFirst() | ||
| dropped += 1 |
There was a problem hiding this comment.
Reset the merge baseline when evicting queued heartbeats
When an outage fills this queue, inFlight remains the original oldest heartbeat while these lines discard the heartbeats immediately following it. After recovery, send(_:) computes its pulsetime from that stale lastSent to the oldest retained heartbeat and, on a data change, refreshes the old event across the entire gap. Consequently, hours of deliberately dropped history can be attributed to the window active before the outage rather than omitted; record the discontinuity when evicting and prevent merging across it.
Useful? React with 👍 / 👎.
🤖 AI code reviewReplaces the fire-and-forget per-heartbeat Task with a serialized HeartbeatSender queue that retries failed sends with exponential backoff, computes pulsetimes at send time relative to the last accepted heartbeat, and creates the bucket lazily before the first send. Removes the global oldHeartbeat variable and the eager createBucket() call in start(). Safe to merge — no P0/P1 findingsConfidence 5/5 ✅ No thread-worthy findings. Advisory notes follow; they are retained without opening review threads. 1 advisory finding (summary-only, not scored)These P2 guard, heuristic, trade-off, or documentation claims are retained for judgment without opening review threads.
This is a How this was verified: static preflight: fix-commit + touched-files scan (rule 7) Files changed (1) — the diff as I read it
Previous review passes
Reviewed Maintainer commands
|
|
This makes the Swift helper crash on its first heartbeat: |
|
Confirmed. On current master v0.14.1 is not affected (its pin predates #162). The current |
…172) Since #162, `heartbeatSender` is a top-level `let` declared after the top-level `start()` and `RunLoop.main.run()` calls. Top-level globals in the helper are initialized only when execution reaches them, and the run loop never returns, so the first heartbeat used an uninitialized HeartbeatSender and the helper crashed with SIGSEGV right after launch. Nothing was recorded with the default macOS strategy. Move startup to the end of the file so every global is initialized first.
The pinned aw-watcher-window (10638b2, #168) is one commit past the regression introduced by ActivityWatch/aw-watcher-window#162: the Swift helper declares `heartbeatSender` after the top-level `start()` / `RunLoop.main.run()`, so the first heartbeat hits an uninitialized sender and the helper exits 139 (SIGSEGV in HeartbeatSender.enqueue). Both candidate repairs merged on 2026-11-04: - ActivityWatch/aw-watcher-window#172 (86c5203) - move startup to the end of macos.swift - ActivityWatch/aw-watcher-window#146 (12000f5) - keep foreground tracking when the browser context fails; fixes the same ordering 12000f5 is aw-watcher-window master, contains 86c5203 (compare: ahead 2, behind 0), and is a fast-forward of the current pin. v0.14.1 pins c8814e4978 (pre-#162) and is unaffected. A v0.14.2 tag at the current pin ships a macOS window watcher that crashes on start, so this needs to land before the tag. Git-Session-Id: 91c15f3b-345b-4a65-8679-c98b93dddbac
Fixes #160.
Problem
sendHeartbeatspawned a newTaskfor every poll (10 s) and every focus change, nothing serialized them, and a failed send was logged and dropped. During the aw-server stall in ActivityWatch/aw-server#187 six heartbeats were in flight at once, and becauseoldHeartbeatwas only updated on success, the next successful send carried a pulsetime of several minutes and credited the whole gap to one window.createBucket()was fire-and-forget too: a watcher that starts before the server (typical at login) never gets its bucket and every heartbeat 404s until restart.Change
HeartbeatSender(anactor) with a bounded in-memory queue (10 000 entries, ~28 h at one per 10 s; oldest dropped past that, counted and logged).Verification
No test target exists for the Swift binary, so I built a harness from this file's sender code (structs, logging,
createBucket,HeartbeatSender,sendHeartbeatSingle) that enqueues a heartbeat every 0.5 s for 12 s, alternating window every 3 s, against a port where an aw-server-rust is started 5 s in:Bucket afterwards (oldest first):
AppA win 03.0 s,AppB win 13.0 s,AppA win 23.0 s,AppB win 32.5 s. Four events, 11.6 s total, i.e. every heartbeat delivered in order and merged as if the server had been up all along.swiftc -target arm64-apple-macosx12.0builds clean apart from two pre-existing optional-interpolation warnings.Not in this PR