Skip to content

fix(logging): dedup and back off repeating poll exceptions - #157

Merged
ErikBjare merged 6 commits into
ActivityWatch:masterfrom
TimeToBuildBob:fix/exception-log-flood-78
Oct 6, 2026
Merged

ErikBjare merged 6 commits into
ActivityWatch:masterfrom
TimeToBuildBob:fix/exception-log-flood-78

Conversation

@TimeToBuildBob

Copy link
Copy Markdown
Contributor

Refs #78, ActivityWatch/activitywatch#629.

A persistent error in get_current_window wrote a full traceback every poll, which filled disks (160 GB in #78). Now the traceback is logged once per distinct (type, message), a one-line summary follows every 100 identical repeats, and polling backs off exponentially up to 60s. The streak resets on the first successful poll.

Note: BrokenPipeError is an OSError, which the loop already treats as fatal, so the pure-BrokenPipe case no longer spins; this guards the remaining non-OSError Xlib/other failures.

Stacked on #155 (same except-block in heartbeat_loop); the diff against master includes #155's commits until it merges. Only the last commit is new. Tests added in tests/test_main.py.

@greptile-apps

greptile-apps Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

[Medium risk] Adds deduplication and backoff for repeating poll errors.

The PR appears safe to merge; no new actionable issue or outstanding previous finding remains.

Summary

The PR limits repeated poll-error logging and adds bounded retry backoff, with tests for repeated errors, alternating errors, and long poll intervals.

Reviews (6) · Last reviewed commit: "fix(logging): bound total retry interval..."

Comment thread aw_watcher_window/main.py Resolved
Comment thread aw_watcher_window/main.py Resolved
Comment thread aw_watcher_window/main.py Resolved
@TimeToBuildBob

TimeToBuildBob commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor Author

🤖 AI code review

Adds Wayland detection warning and X display-connection error handling to heartbeat_loop, plus deduplication and exponential backoff for repeated poll exceptions. Moves the DISPLAY check after setup_logging and adds a Wayland warning before it. Adds tests for the new warning and backoff behavior.

Safe to merge — no P0/P1 findings

Confidence 5/5

✅ No findings. The diff looks correct to me on this pass.

Files changed (2) — the diff as I read it
  • aw_watcher_window/main.py — Adds _is_xconn_error, _warn_wayland_once, and backoff/dedup state in heartbeat_loop; moves DISPLAY check after logging and adds Wayland warning.
  • tests/test_main.py — Adds tests for Wayland warning and heartbeat_loop backoff/dedup behavior.
Previous review passes
commit score findings engine when
7a552cd072b0 3/5 1 llm 2026-10-02 08:23 UTC
d30ccb43d3eb 3/5 2 llm 2026-10-02 15:30 UTC

Reviewed 1aea2bae029a · openrouter/deepseek/deepseek-v4-flash-0731 · llm engine · 67s · about this reviewer

Maintainer commands

@TimeToBuildBob review (own line) — fresh review · @TimeToBuildBob fix — a worker acts on the findings. Once per comment; 👀 = received.

Comment thread aw_watcher_window/main.py Resolved
@TimeToBuildBob

Copy link
Copy Markdown
Contributor Author

@greptileai review

Comment thread aw_watcher_window/main.py Fixed
Comment thread aw_watcher_window/main.py Resolved
Comment thread aw_watcher_window/main.py Resolved
Comment thread aw_watcher_window/main.py Resolved
@TimeToBuildBob

Copy link
Copy Markdown
Contributor Author

@greptileai review

Comment thread aw_watcher_window/main.py Resolved
@TimeToBuildBob

Copy link
Copy Markdown
Contributor Author

CI-green and mergeable (Greptile 5/5) — waiting only on a maintainer click.

This PR is ready to merge, but the bot has pull-only access to this repo and can't self-merge — surfacing it here so it isn't lost. The monitoring loop will stop re-flagging it now that this note is posted.

Comment thread aw_watcher_window/main.py Resolved
Comment thread tests/test_main.py Resolved
@TimeToBuildBob

Copy link
Copy Markdown
Contributor Author

@greptileai review

Comment thread aw_watcher_window/main.py Resolved
Comment thread tests/test_main.py Resolved
@TimeToBuildBob

Copy link
Copy Markdown
Contributor Author

@greptileai review

TimeToBuildBob and others added 6 commits October 6, 2026 21:10
A persistent error in get_current_window logged a full traceback every
poll (160 GB of logs in ActivityWatch#78). Log the traceback once per distinct error,
then a one-line summary every 100 repeats, and back off up to 60s.

Refs ActivityWatch#78, ActivityWatch/activitywatch#629

Git-Session-Id: a97b
…, long poll_time crashes

Two P1 findings from Greptile review:

1. Alternating errors evade backoff and deduplication: the old code tracked only
   the immediately preceding error signature (_last_error), so two errors
   alternating A→B→A→B each looked "new" — full traceback logged every poll and
   _error_repeats reset to 0, leaving no backoff.  Replace with _seen_errors (a
   set of all signatures seen in the current failure streak) so the repeat counter
   accumulates regardless of which distinct error fires each poll.

2. poll_time > 60s crashes the watcher: the backoff sleep
   `min(poll_time * 2^n, 60.0) - poll_time` goes negative when poll_time exceeds
   the 60s cap, raising ValueError in sleep().  Wrap with max(0.0, ...).

Tests added for both cases.

Git-Session-Id: 40e1039a-2160-5f7b-950a-b18c8a2bec93
Two issues addressed per Greptile P1 review:
- _seen_errors set was unbounded: with a rotating error message (e.g. X
  window IDs in the exception text) each poll added a new signature and
  logged a full traceback, undoing the dedup goal. Cap at _MAX_SEEN_ERRORS
  (50); once the cap is reached, unknown signatures are treated as recurring
  (suppress traceback, emit periodic summary).
- Backoff fired on the first transient error: with poll_time=5s the extra
  sleep made the next poll ~10s away, exceeding the 7.5s heartbeat pulsetime
  and risking a gap in recorded activity. Change condition to _error_repeats > 1
  so the extra sleep only kicks in for sustained (2+) failures.

Git-Session-Id: 4bc05df1-0164-557f-bab1-1be84a7b8d5b
…odeQL

CodeQL flagged the bare 'except OSError: pass' inside the fatal-error handler
(around the logger.exception call) as having no explanatory comment. Added a
comment clarifying that the OSError is intentionally swallowed: logging can
raise OSError ([Errno 5] Input/output error) when stdout is closed, and we
need execution to reach the break statement below.

Git-Session-Id: 4bc05df1-0164-557f-bab1-1be84a7b8d5b
For poll_time > 60s the old formula min(poll_time * 2^n, 60) - poll_time
produced a negative value clamped to 0.0, so no extra backoff occurred.
Replace with min(poll_time * (2^n - 1), 60.0) — the extra sleep is now
independent of poll_time and always non-negative.

Update test assertions to reflect the new cap (max extra sleep = 60s, not 59s)
and add a positive-backoff assertion for the large-poll_time test case so the
test is no longer vacuous.

Git-Session-Id: c0c8
…stretches long poll_time

Git-Session-Id: 56d77c68-d50a-5ab0-b834-2e6c7fdcc594
@TimeToBuildBob
TimeToBuildBob force-pushed the fix/exception-log-flood-78 branch from 1aea2ba to 0d3ccf5 Compare October 6, 2026 21:10
@TimeToBuildBob

Copy link
Copy Markdown
Contributor Author

Rebased onto master now that #155 is merged. The PR contains only its own 6 commits (0d3ccf5). Resolved one conflict in tests/test_main.py: kept #155's final assertions for the X-auth test (≤1s sleep chunks, 7s total) and added this PR's repeated-exception test. 107 tests pass locally. mypy output is identical to master (14 pre-existing platform-stub errors).

@ErikBjare

Copy link
Copy Markdown
Member

@greptileai review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants