Skip to content

fix(macos): serialize heartbeats through a retrying queue, create the bucket lazily - #162

Merged
ErikBjare merged 2 commits into
masterfrom
macos-heartbeat-queue
Oct 10, 2026
Merged

ErikBjare merged 2 commits into
masterfrom
macos-heartbeat-queue

Conversation

@ErikBjare

Copy link
Copy Markdown
Member

Fixes #160.

Problem

sendHeartbeat spawned a new Task for every poll (10 s) and every focus change, nothing serialized them, and a failed send was logged and dropped. During the aw-server stall in ActivityWatch/aw-server#187 six heartbeats were in flight at once, and because oldHeartbeat was only updated on success, the next successful send carried a pulsetime of several minutes and credited the whole gap to one window. createBucket() was fire-and-forget too: a watcher that starts before the server (typical at login) never gets its bucket and every heartbeat 404s until restart.

Change

  • HeartbeatSender (an actor) with a bounded in-memory queue (10 000 entries, ~28 h at one per 10 s; oldest dropped past that, counted and logged).
  • One request in flight, strictly in order. A failed heartbeat stays at the head and is retried with exponential backoff, 1 s doubling to 60 s; a success resets the delay.
  • Pulsetimes are computed at send time relative to the last heartbeat the server accepted, so the existing "send the old heartbeat for merging" semantics (fix: misc fixes to swift script on macos, incl URL inspection #69) are unchanged and still cover any gap introduced by the delay.
  • The bucket is created (or confirmed with 304) by the sender before its first send, with the same retry, so a server that comes up later gets the bucket and then the backlog.
  • The sub-second dedupe for resize storms is kept at enqueue time.

Verification

No test target exists for the Swift binary, so I built a harness from this file's sender code (structs, logging, createBucket, HeartbeatSender, sendHeartbeatSingle) that enqueues a heartbeat every 0.5 s for 12 s, alternating window every 3 s, against a port where an aw-server-rust is started 5 s in:

18:50:09 Failed to send heartbeat (1 queued), retrying in 1s: Could not connect to the server.
18:50:10 Failed to send heartbeat (3 queued), retrying in 2s: ...
18:50:12 Failed to send heartbeat (7 queued), retrying in 4s: ...
18:50:21 enqueued 24 heartbeats, waiting for the queue to drain

Bucket afterwards (oldest first): AppA win 0 3.0 s, AppB win 1 3.0 s, AppA win 2 3.0 s, AppB win 3 2.5 s. Four events, 11.6 s total, i.e. every heartbeat delivered in order and merged as if the server had been up all along.

swiftc -target arm64-apple-macosx12.0 builds clean apart from two pre-existing optional-interpolation warnings.

Not in this PR

  • Persisting the queue to disk across watcher restarts (the Python watchers get that from aw-client's persist-queue); the TODO about letting the Python wrapper own the sending still stands.

… bucket lazily

Every poll and focus change spawned its own Task: when the server was slow
they piled up in parallel (six in flight observed during an aw-server stall)
and a failed heartbeat was dropped, so the time until the next successful
send was credited to whatever window was sent next. Bucket creation was
fire-and-forget too, so a watcher started before the server (login) never
got a bucket and every heartbeat 404ed.

HeartbeatSender is an actor with a bounded in-memory queue: one request in
flight, in order, failed heartbeats retried with exponential backoff (1 s
to 60 s), pulsetimes computed at send time relative to the last accepted
heartbeat so the server-side merge still covers the gap, and the bucket
created (or confirmed) before the first send with the same retry.

Verified with a harness built from this file's sender code against an
aw-server-rust started 5 s after the watcher: 24 heartbeats over 12 s,
alternating windows every 3 s, retried at 1/2/4 s, all delivered in
order and merged into 4 events of 3 s (last 2.5 s).

Fixes #160.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-10T16:59:49.046576Z 2404d56 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 28094d04fe

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread aw_watcher_window/macos.swift Outdated
let heartbeatSender = HeartbeatSender()

func sendHeartbeat(_ heartbeat: Heartbeat) {
Task { await heartbeatSender.enqueue(heartbeat) }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve producer order before enqueueing

Each call launches an independent unstructured Task, and Swift does not guarantee that these tasks submit their actor calls in creation order. With back-to-back window callbacks, a newer heartbeat can therefore reach enqueue first; when the older one arrives, its negative interval relative to lastEnqueued satisfies <= 0.5 and it is discarded. Feed the actor through a single ordered producer or another synchronously ordered mechanism so the new sender actually preserves callback chronology.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2404d56: the sender is now a lock-protected class and enqueue runs synchronously on the calling (main) thread, so call order is delivery order; only the drain runs in a Task.

Comment thread aw_watcher_window/macos.swift Outdated
Comment on lines +463 to +465
if queue.count >= Self.maxQueued {
queue.removeFirst()
droppedSinceLastLog += 1

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid removing the in-flight heartbeat from the bounded queue

When the queue is full while drain is suspended in createBucket or send, actor reentrancy allows this removal to delete the head already captured as next. If that request succeeds, drain subsequently calls queue.removeFirst() again and deletes the following heartbeat without sending it. This is especially plausible when recovering with a full backlog; keep the in-flight item separate from the pending queue or remove it by verified identity after success.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2404d56: the heartbeat being sent is taken out of the queue into a separate in-flight slot and only cleared on success; dropping the oldest pending entry can no longer touch it. Harness re-run gives the same 4 merged events / 11.6 s.

@greptile-apps

greptile-apps Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; both previous findings are fixed and no new blocking issue remains.

Summary

The PR queues macOS heartbeats, retries failed requests, and creates the bucket before sending.

  • The macOS watcher sends heartbeats in order and retries failures.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Poll or focus change] --> B[Add heartbeat under lock]
  B --> C[Pending queue]
  C --> D[Move oldest heartbeat into inFlight]
  D --> E[Create bucket if needed]
  E --> F[Send heartbeat]
  F -->|Success| G[Clear inFlight]
  G --> C
  E -->|Failure| H[Wait with backoff]
  F -->|Failure| H
  H --> E
Loading

Reviews (2) · Last reviewed commit: "fix(macos): enqueue synchronously and ke..." · Reviewed by Greptile

Comment thread aw_watcher_window/macos.swift Outdated
bucketReady = true
}
try await send(next)
queue.removeFirst()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Recovery drops an unsent heartbeat

When the queue is full, enqueue can remove the heartbeat that drain is currently sending while send(next) waits for the server. After that send succeeds, this removeFirst() removes a second heartbeat that was never sent. Recovery after a long outage can therefore lose window changes beyond the intended oldest-entry drops.

Keep the in-flight heartbeat separate from the queue, or remove only the entry that actually finished sending.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already addressed in 2404d56 (pushed just before this review): enqueue is synchronous on the calling thread under a lock, so callback order is delivery order, and the heartbeat being sent lives in a separate in-flight slot that a full-queue drop cannot touch.

Comment on lines +523 to +525
func sendHeartbeat(_ heartbeat: Heartbeat) {
Task { await heartbeatSender.enqueue(heartbeat) }
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Tasks can reorder heartbeats

sendHeartbeat still launches an independent Task for each callback. The actor receives heartbeats in task scheduling order, not necessarily callback order. If a later heartbeat reaches enqueue first, the earlier heartbeat has a negative time difference and is discarded by the <= 0.5 check—even when the observations were more than half a second apart. That loses a window change and leaves the promised ordering fix incomplete.

Put heartbeats into one ordered stream before the asynchronous handoff.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already addressed in 2404d56 (pushed just before this review): enqueue is synchronous on the calling thread under a lock, so callback order is delivery order, and the heartbeat being sent lives in a separate in-flight slot that a full-queue drop cannot touch.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update. I rechecked 2404d56: sendHeartbeat now calls heartbeatSender.enqueue synchronously, and enqueue serializes queue mutation under NSLock, so callbacks on the main thread preserve delivery order before the asynchronous drain begins. The separate inFlight slot also prevents a full pending queue from dropping the heartbeat currently being retried. This resolves my ordering and in-flight queue concerns.

…t of the bounded queue

Review follow-ups on #162: reaching the actor through a detached Task per
call did not guarantee that back-to-back callbacks arrived in order, and
an older heartbeat arriving second was dropped by the sub-second dedupe;
and dropping the oldest pending entry while a send was suspended could
remove the very heartbeat being sent. The sender is now a lock-protected
class whose enqueue runs synchronously on the caller (main thread), and
the heartbeat being retried is held apart from the pending queue.
@ErikBjare

Copy link
Copy Markdown
Member Author

@greptileai review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2404d5638a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +476 to +478
if queue.count >= Self.maxQueued {
queue.removeFirst()
dropped += 1

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reset the merge baseline when evicting queued heartbeats

When an outage fills this queue, inFlight remains the original oldest heartbeat while these lines discard the heartbeats immediately following it. After recovery, send(_:) computes its pulsetime from that stale lastSent to the oldest retained heartbeat and, on a data change, refreshes the old event across the entire gap. Consequently, hours of deliberately dropped history can be attributed to the window active before the outage rather than omitted; record the discontinuity when evicting and prevent merging across it.

Useful? React with 👍 / 👎.

@TimeToBuildBob

TimeToBuildBob commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

🤖 AI code review

Replaces the fire-and-forget per-heartbeat Task with a serialized HeartbeatSender queue that retries failed sends with exponential backoff, computes pulsetimes at send time relative to the last accepted heartbeat, and creates the bucket lazily before the first send. Removes the global oldHeartbeat variable and the eager createBucket() call in start().

Safe to merge — no P0/P1 findings

Confidence 5/5

✅ No thread-worthy findings. Advisory notes follow; they are retained without opening review threads.

1 advisory finding (summary-only, not scored)

These P2 guard, heuristic, trade-off, or documentation claims are retained for judgment without opening review threads.

⚠️ P2 medium — aw_watcher_window/macos.swift

This is a fix(...) PR but no test files are included in the diff. Erik's feedback: 'where is the repro & fixes they are supposed to catch' (gptme#3441), 'that measurement should come with a regression test' (gptme#3446). Add a test that would have caught this bug. (Advisory: Erik merged all such PRs but consistently requested tests.)

Add a test file that reproduces the bug before the fix and passes after it.

How this was verified: static preflight: fix-commit + touched-files scan (rule 7)

Files changed (1) — the diff as I read it
  • aw_watcher_window/macos.swift — Adds HeartbeatSender class with a locked queue, in-flight retry, backoff, and lazy bucket creation; removes oldHeartbeat global and eager createBucket call.
Previous review passes
commit score findings engine when
2404d5638a89 5/5 0 llm 2026-10-10 17:33 UTC

Reviewed 2404d5638a89 · openrouter/deepseek/deepseek-v4-flash-0731 · llm engine · 123s · about this reviewer

Maintainer commands

@TimeToBuildBob review (own line) — fresh review · @TimeToBuildBob fix — a worker acts on the findings. Once per comment; 👀 = received.

@andreivince

Copy link
Copy Markdown
Contributor

This makes the Swift helper crash on its first heartbeat: heartbeatSender is declared after the top-level start() and RunLoop.main.run(), so it's never initialized (SIGSEGV in HeartbeatSender.enqueue, exit status 139). Fix in #172.

@TimeToBuildBob

Copy link
Copy Markdown
Contributor

Confirmed. On current master start() / RunLoop.main.run() still run at lines 211–212, and let heartbeatSender = HeartbeatSender() is at 615, so the first heartbeat uses an uninitialized sender (SIGSEGV in HeartbeatSender.enqueue). #172 is the right fix.

v0.14.1 is not affected (its pin predates #162). The current activitywatch master pin is, because it includes #162.

ErikBjare pushed a commit that referenced this pull request Nov 4, 2026
…172)

Since #162, `heartbeatSender` is a top-level `let` declared after the
top-level `start()` and `RunLoop.main.run()` calls. Top-level globals in
the helper are initialized only when execution reaches them, and the run
loop never returns, so the first heartbeat used an uninitialized
HeartbeatSender and the helper crashed with SIGSEGV right after launch.
Nothing was recorded with the default macOS strategy. Move startup to the
end of the file so every global is initialized first.
ErikBjare pushed a commit to ActivityWatch/activitywatch that referenced this pull request Nov 6, 2026
The pinned aw-watcher-window (10638b2, #168) is one commit past the
regression introduced by ActivityWatch/aw-watcher-window#162: the Swift
helper declares `heartbeatSender` after the top-level `start()` /
`RunLoop.main.run()`, so the first heartbeat hits an uninitialized sender
and the helper exits 139 (SIGSEGV in HeartbeatSender.enqueue).

Both candidate repairs merged on 2026-11-04:

- ActivityWatch/aw-watcher-window#172 (86c5203) - move startup to the end
  of macos.swift
- ActivityWatch/aw-watcher-window#146 (12000f5) - keep foreground tracking
  when the browser context fails; fixes the same ordering

12000f5 is aw-watcher-window master, contains 86c5203 (compare: ahead 2,
behind 0), and is a fast-forward of the current pin. v0.14.1 pins
c8814e4978 (pre-#162) and is unaffected.

A v0.14.2 tag at the current pin ships a macOS window watcher that crashes
on start, so this needs to land before the tag.

Git-Session-Id: 91c15f3b-345b-4a65-8679-c98b93dddbac
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

macOS (Swift): heartbeats are unserialized fire-and-forget tasks; pile up in parallel when the server is slow, failures are dropped

3 participants