Repository navigation
fix(query): cap events one query timeperiod may load (query_max_events) - #226
Conversation
A long-range query (All time, Year) loads every event in range into Python lists at ~1.3 KB each, with no bound. On a large database that is gigabytes, the machine swaps, and heartbeats stall long enough to hit the 10 s heartbeat lock timeout (503). The query also keeps running after the client times out, so retries stack. Count matching rows before each query_bucket fetch and fail the timeperiod with a QueryException (HTTP 400) once it would load more than query_max_events (default 1,000,000, about 1.3 GB). The client gets an immediate, actionable error telling it to split the period. Set 0 to disable. Refs ActivityWatch#187 Git-Session-Id: 3127bce9-b261-4bfa-8d67-6b5fc0e63453
Review guide (auto-posted)Structured warm-up for reviewers — what changed, what to run, where to look. Key files
Suggested verification
Known risks / watch points
Suggested review focus
Generated by |
[Medium impact] The PR appears safe to merge, with no outstanding findings. SummaryAdds
Diagram%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Query timeperiod] --> B[Create event budget]
B --> C[Count matching events]
C --> D{Within remaining budget?}
D -->|Yes| E[Charge budget and fetch events]
E --> C
D -->|No| F[Raise QueryException and return HTTP 400]
Reviews (3) · Last reviewed commit: "Revert "test(query): assert the event li..." · Reviewed by Greptile |
Spy on the storage fetch so moving the budget check after get() fails the test instead of silently reintroducing the memory spike. Git-Session-Id: d9d02e20-31d1-5d01-ab54-682c19b173c2
Git-Session-Id: 3127bce9-b261-4bfa-8d67-6b5fc0e63453
This comment has been minimized.
This comment has been minimized.
This reverts commit 616f238. Git-Session-Id: 3127bce9-b261-4bfa-8d67-6b5fc0e63453
|
CI-green and mergeable (Greptile 5/5) — waiting only on a maintainer click. This PR is ready to merge, but the bot has pull-only access to this repo and can't self-merge — surfacing it here so it isn't lost. The monitoring loop will stop re-flagging it now that this note is posted. |
|
@greptileai review |
🤖 AI code reviewSafe to merge — no P0/P1 findings on latest reviewUpdated after inline dispositions on finding threads — this is the current state; the verdict below is frozen at review time and is kept as the historical record of that pass. Finding disposition
Adds an event-count budget to query2 so each timeperiod can load at most query_max_events events, failing with a QueryException when exceeded. Adds a config option query_max_events (default 1,000,000, 0 disables) threaded through main, AWFlask, and ServerAPI, plus a new test module covering the limit, per-period behavior, and HTTP 400 mapping. Needs a look — P2 onlyConfidence 4/5 1 finding ·
|
|
Merge recommendation, checked on head
This is waiting on a maintainer merge. |
Part of #187: stops one long-range query from taking the server, and with it heartbeats, down.
Change
Before each
query_bucketfetch, count the matching rows. Once one query timeperiod would load more thanquery_max_eventsevents (summed over all itsquery_bucketcalls), fail that timeperiod with aQueryException, which is a 400 with this message:1000000(measured ~1.3 KB per loaded event, so about 1.3 GB).0disables it. Configurable viaquery_max_eventsinaw-server.toml._where_rangefilter as the fetch. It costs nothing measurable on a query under the limit: 11.44/11.48 s with this PR vs 11.52/11.32 s on master, for 310k events.Measurements
Synthetic peewee DB, no real user data: 200k window events, 10k afk, 100k web over 365 days (58 MB). The harness has three 1 Hz heartbeat senders, like window/afk/web watchers, plus three concurrent "Year" Activity-style queries (flood,
filter_period_intersect,categorize,merge_events_by_keys) with a 30 s client timeout. The server runs undermemory.high=400Mto reproduce the memory pressure from the incident, which was swapping.query_max_events=100000, scaled to the smaller DB)A second master-equivalent run, where the limit did not trip, reproduced the outage: 30 heartbeats over 10 s, two 503s, and the server busy until 359 s.
Default limit on an "All time" DB (1.2M window + 60k afk + 600k web over 1100 days, 351 MB): the query gets a 400 in 0.57 s, RSS stays at 51 MB, and heartbeats are unaffected. On master the same query would load ~1.86M events, about 2.5 GB at the measured per-event cost.
Without memory pressure, master heartbeats survive even six concurrent abandoned queries (max latency 0.83 s, RSS 1.8 GB). The GIL alone does not cause the 503s; memory exhaustion does. That is why this PR bounds memory instead of changing the heartbeat lock.
Not in this PR
Eventobjects (point 1 at the root). That is an aw-core change and a bigger one.Tests
tests/test_query_event_limit.py: within the limit, sum across buckets, per-timeperiod scope,0disables, and the HTTP 400 path. Local results:tests/pass excepttest_client.py, which needs a running server and fails the same way on master. mypy and ruff are clean, and black is applied.