Skip to content

fix(query): cap events one query timeperiod may load (query_max_events) - #226

Merged
ErikBjare merged 4 commits into
ActivityWatch:masterfrom
TimeToBuildBob:fix/issue-187-heartbeat-starvation
Nov 17, 2026
Merged

ErikBjare merged 4 commits into
ActivityWatch:masterfrom
TimeToBuildBob:fix/issue-187-heartbeat-starvation

Conversation

@TimeToBuildBob

Copy link
Copy Markdown
Contributor

Part of #187: stops one long-range query from taking the server, and with it heartbeats, down.

Change

Before each query_bucket fetch, count the matching rows. Once one query timeperiod would load more than query_max_events events (summed over all its query_bucket calls), fail that timeperiod with a QueryException, which is a 400 with this message:

Query would load N events (bucket 'X' alone has M in START/END), more than the server limit of 1000000 per timeperiod. Split the query into shorter timeperiods, or raise query_max_events in the aw-server config.

  • Default 1000000 (measured ~1.3 KB per loaded event, so about 1.3 GB). 0 disables it. Configurable via query_max_events in aw-server.toml.
  • The limit applies per timeperiod, not per request, so views that split a range into days (most aw-webui views) are unaffected.
  • The COUNT uses the same _where_range filter as the fetch. It costs nothing measurable on a query under the limit: 11.44/11.48 s with this PR vs 11.52/11.32 s on master, for 310k events.

Measurements

Synthetic peewee DB, no real user data: 200k window events, 10k afk, 100k web over 365 days (58 MB). The harness has three 1 Hz heartbeat senders, like window/afk/web watchers, plus three concurrent "Year" Activity-style queries (flood, filter_period_intersect, categorize, merge_events_by_keys) with a 30 s client timeout. The server runs under memory.high=400M to reproduce the memory pressure from the incident, which was swapping.

master this PR (query_max_events=100000, scaled to the smaller DB)
query result client timeout at 30 s; server kept computing until 372 s 400 in 0.09–0.12 s
server peak RSS 407 MB (pinned at memory.high, rest reclaimed/swapped) 52 MB
heartbeats > 10 s 36 of 398 0 of 33
heartbeat 503s 5 (first at 28 s, last at 205 s) 0
max heartbeat latency 28.9 s 0.08 s

A second master-equivalent run, where the limit did not trip, reproduced the outage: 30 heartbeats over 10 s, two 503s, and the server busy until 359 s.

Default limit on an "All time" DB (1.2M window + 60k afk + 600k web over 1100 days, 351 MB): the query gets a 400 in 0.57 s, RSS stays at 51 MB, and heartbeats are unaffected. On master the same query would load ~1.86M events, about 2.5 GB at the measured per-event cost.

Without memory pressure, master heartbeats survive even six concurrent abandoned queries (max latency 0.83 s, RSS 1.8 GB). The GIL alone does not cause the 503s; memory exhaustion does. That is why this PR bounds memory instead of changing the heartbeat lock.

Not in this PR

Tests

tests/test_query_event_limit.py: within the limit, sum across buckets, per-timeperiod scope, 0 disables, and the HTTP 400 path. Local results: tests/ pass except test_client.py, which needs a running server and fails the same way on master. mypy and ruff are clean, and black is applied.

A long-range query (All time, Year) loads every event in range into
Python lists at ~1.3 KB each, with no bound. On a large database that is
gigabytes, the machine swaps, and heartbeats stall long enough to hit the
10 s heartbeat lock timeout (503). The query also keeps running after
the client times out, so retries stack.

Count matching rows before each query_bucket fetch and fail the
timeperiod with a QueryException (HTTP 400) once it would load more
than query_max_events (default 1,000,000, about 1.3 GB). The client gets
an immediate, actionable error telling it to split the period. Set 0 to
disable.

Refs ActivityWatch#187

Git-Session-Id: 3127bce9-b261-4bfa-8d67-6b5fc0e63453
@TimeToBuildBob

Copy link
Copy Markdown
Contributor Author

Review guide (auto-posted)

Structured warm-up for reviewers — what changed, what to run, where to look.

Key files

  • tests/test_query_event_limit.py (+105/-0, Δ105)
  • aw_server/api.py (+74/-3, Δ77)
  • aw_server/server.py (+10/-2, Δ12)
  • aw_server/main.py (+5/-0, Δ5)
  • aw_server/config.py (+4/-0, Δ4)

Suggested verification

  • uv run pytest tests/test_query_event_limit.py -q
  • make typecheck

Known risks / watch points

  • No automatic high-risk tags; use file list + diff for judgment.

Suggested review focus

  • Confirm behavior matches the PR description acceptance criteria.
  • Skim the largest diffs first (listed above).
  • If CI is green, spot-check the highest-risk paths called out here.

Generated by scripts/github/pr-warmup-review-guide.py for #226.

@greptile-apps

greptile-apps Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge, with no outstanding findings.

Summary

Adds query_max_events to cap how many events each query timeperiod loads.

  • Long queries stop before a timeperiod loads too many events.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Query timeperiod] --> B[Create event budget]
    B --> C[Count matching events]
    C --> D{Within remaining budget?}
    D -->|Yes| E[Charge budget and fetch events]
    E --> C
    D -->|No| F[Raise QueryException and return HTTP 400]
Loading

Reviews (3) · Last reviewed commit: "Revert "test(query): assert the event li..." · Reviewed by Greptile

Comment thread tests/test_query_event_limit.py Resolved
Spy on the storage fetch so moving the budget check after get() fails the
test instead of silently reintroducing the memory spike.

Git-Session-Id: d9d02e20-31d1-5d01-ab54-682c19b173c2
Git-Session-Id: 3127bce9-b261-4bfa-8d67-6b5fc0e63453
@TimeToBuildBob

This comment has been minimized.

This reverts commit 616f238.

Git-Session-Id: 3127bce9-b261-4bfa-8d67-6b5fc0e63453
@TimeToBuildBob

Copy link
Copy Markdown
Contributor Author

CI-green and mergeable (Greptile 5/5) — waiting only on a maintainer click.

This PR is ready to merge, but the bot has pull-only access to this repo and can't self-merge — surfacing it here so it isn't lost. The monitoring loop will stop re-flagging it now that this note is posted.

@TimeToBuildBob

Copy link
Copy Markdown
Contributor Author

@greptileai review

@TimeToBuildBob

TimeToBuildBob commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor Author

🤖 AI code review

Safe to merge — no P0/P1 findings on latest review

Updated after inline dispositions on finding threads — this is the current state; the verdict below is frozen at review time and is kept as the historical record of that pass.

Finding disposition
Finding Severity State
tests/test_query_event_limit.py:66 P2 rejected

Adds an event-count budget to query2 so each timeperiod can load at most query_max_events events, failing with a QueryException when exceeded. Adds a config option query_max_events (default 1,000,000, 0 disables) threaded through main, AWFlask, and ServerAPI, plus a new test module covering the limit, per-period behavior, and HTTP 400 mapping.

Needs a look — P2 only

Confidence 4/5

1 finding · ⚠️ 1 P2

⚠️ P2 medium — tests/test_query_event_limit.py:66

The test test_limit_counts_all_buckets_of_one_period relies on the memory storage strategy's get_events method being called once for bucket 'a' and then the exception being raised before bucket 'b' is fetched. This is correct for the current aw_query execution order, but the test does not assert that the second query_bucket call ever consulted the budget; if aw_query were to evaluate query_bucket('b') eagerly before query_bucket('a') or if the storage spy did not see the call, the assertion could still pass. In practice the test does pin the intended fail-fast behavior, so this is a minor robustness note rather than a defect.

assert fetched == []

How this was verified: Traced _BudgetedBucket.get in aw_server/api.py lines 131-136: charge runs before the call to self._bucket.get. Therefore the spy on storage.get_events (which wraps the real get) is never invoked for any bucket when the first charge raises.

Consensus: 2/3 passes agreed — pass 3 looked and disagreed
Distinct keys: 1 (general)

1 advisory finding (summary-only, not scored)

These P2 guard, heuristic, trade-off, or documentation claims are retained for judgment without opening review threads.

⚠️ P2 medium — aw_server/api.py:102

The new _EventBudget and _BudgetedBucket wrappers delegate attribute access via getattr, but they do not implement Python's special/dunder methods. If aw_query's query engine ever calls len(), iteration, or another dunder on the db or bucket object, those calls will not be forwarded to the wrapped Datastore/Bucket and will raise TypeError. In the current aw_query usage this does not occur (query_bucket calls .get), but the wrapper is a partial proxy and any future or alternative query implementation that uses e.g. for event in bucket would break without touching this PR's code.

if limit is None:
            limit = -1
        if 0 <= limit < count:
            count = limit

How this was verified: Compared with ServerAPI.get_events at aw_server/api.py line 418-419, which explicitly handles None by converting to -1 before calling bucket.get. The wrapper does not perform that normalization.

Consensus: 2/3 passes agreed — pass 2 looked and disagreed
Distinct keys: 1 (general)

Files changed (5) — the diff as I read it
  • aw_server/api.py — Adds DEFAULT_QUERY_MAX_EVENTS, _EventBudget and _BudgetedBucket wrappers, and uses them in query2 for each timeperiod.
  • aw_server/config.py — Adds query_max_events to the default [server] and [server-testing] TOML config sections.
  • aw_server/main.py — Reads query_max_events from config and passes it to _start.
  • aw_server/server.py — Adds query_max_events parameter to AWFlask and _start and forwards it to ServerAPI.
  • tests/test_query_event_limit.py — Adds tests for the event limit: within limit, per-period accounting, disable via zero, and 400 response.

Reviewed 36b5d1905330 · openrouter/deepseek/deepseek-v4-flash-0731 · llm engine · 123s · about this reviewer

Maintainer commands

@TimeToBuildBob review (own line) — fresh review · @TimeToBuildBob fix — a worker acts on the findings. Once per comment; 👀 = received.

Comment thread tests/test_query_event_limit.py Resolved
@TimeToBuildBob

Copy link
Copy Markdown
Contributor Author

Merge recommendation, checked on head 36b5d19

  • Fixed: Greptile's P2 ("tests miss early-fetch regressions") was handled in db2d86e. The spy on get_events asserts that the rejected bucket is never fetched, and that thread is resolved.
  • Dismissed: the AI-review P2 on tests/test_query_event_limit.py:66. If the bucket order flipped, or the spy saw no calls, the test would fail rather than pass. The suggested fetched == [] is also wrong, because bucket a fits the budget. The reasoning is on the thread.
  • Not acted on: the advisory that the _BudgetedBucket proxy is partial. aw_query's query_bucket only calls .get(starttime=, endtime=), so limit stays at its default of -1 and no dunder methods are needed.
  • CI: all green on macOS, Ubuntu and Windows, plus typecheck, lint, format and CodeQL. Greptile is 5/5.
  • Domain risk: the default query_max_events is 1,000,000 per timeperiod. A maintainer may want to run a long-range query (for example a year-long dashboard) against a large real database to confirm the new 400 doesn't hit normal usage.
  • Convergence: no_current_blocking, with 0 P1 and 0 unresolved findings across 1 round (stable_rounds: 0 of 2 required). No further review rounds were triggered.

This is waiting on a maintainer merge.

@ErikBjare
ErikBjare merged commit 869eccf into ActivityWatch:master Nov 17, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants