Repository navigation
Open
Description
Summary
A long-range query from the web UI (All time, then Year) left aw-server (Python, v0.14.0b9) holding 8.6 GB of Python objects. The client had already timed out, but the server kept computing and then spent ~45 minutes freeing the result. Because that teardown holds the GIL, every watcher heartbeat hit the 10 s HeartbeatResource lock timeout and got 503 Server busy for the duration, so tracking stopped.
The server recovered on its own eventually, but it should not get into this state in the first place.
What happened (erb-m2, macOS, 1.9 GB peewee-sqlite.v2.db, 42 buckets)
| time | event |
|---|---|
| 15:52–15:59 | ~500 query2 requests from the web UI (Activity view: All time, which timed out client-side after the configured 120 s, then Year, which errored). aw_transform.flood warnings were the only trace, since successful requests are not logged. |
| ~16:00 | process at 8.6 GB (top: MEM 8606M, CMPRS 8107M), state stuck. One thread spends ~90 % of its samples in frame_dealloc → list_dealloc → dict_dealloc (macOS sample), i.e. a request handler returning and freeing a huge list of event dicts. The machine was swapping (14/15 GB swap used), so each page had to be decompressed to be freed and got recompressed under pressure. 3–30 % CPU, no net progress for 40+ min. |
| 16:04 → 16:37 | Heartbeat lock could not be acquired within timeout, rejecting request + 503 for aw-watcher-window, aw-watcher-afk and aw-watcher-web heartbeats. Watchers queued/retried; the macOS window watcher ended up with 6 parallel in-flight heartbeats (ActivityWatch/aw-watcher-window#160). |
| ~16:45 | teardown finished, process back to 2.9 GB, heartbeats OK again. |
The in-memory QueryCache (#174) is bounded (128 MB of JSON) and is not the cause.
Why
- Query results are fully materialized.
q2_query_bucketcallsdatastore[bucket].get(starttime, endtime)which returns aList[Event]; every transform (flood,filter_period_intersect,categorize, …) builds new lists. A query over a long range with window + afk + browser buckets is millions ofEventdicts, ~1 KB each in Python. There is no bound on result size or on the range a single request may cover. - No cancellation. The web UI aborts the Axios request after
requestTimeout, but Flask/werkzeug keeps running the handler to completion. The user retrying with a smaller range (as I did) stacks a second heavy query on top of the first. - The GIL couples queries and heartbeats. A query thread that is allocating or freeing millions of objects (or page-faulting while doing so) starves the heartbeat thread, which only waits 10 s for its lock before returning 503. Heartbeats are the one thing the server must never drop; they should not share fate with analytics queries.
- Not diagnosable from the log. Only non-2xx responses are logged, so there is no record of which query/period produced the giant result or how long it took.
Proposals (any subset helps)
- Log slow/large query requests at INFO: name, timeperiod(s), duration, result size (events/bytes). Cheap and makes the next report actionable.
- Cancel on client disconnect. Check the request's socket/
environbetween query2 periods (and ideally inside long transforms) and abort when the client has gone away. At minimum, stop iterating thetimeperiodsloop inServerAPI.query2once the client is gone. - Bound work per request. A configurable cap on the time range of one
query_bucket(or on the number of events it may return, erroring with a 4xx that tells the client to split) would turn a 45-minute stall into an immediate, actionable error. aw-webui already splits most views into days; the server should enforce the contract rather than rely on it. - Stream from SQLite instead of materializing where the pipeline allows (peewee
.iterator()), and consider__slots__/lighterEventobjects. Even a 2–3× reduction in per-event memory would have kept this inside RAM. - Isolate heartbeats from queries. Options: run queries in a worker subprocess (multiprocessing pool, so the GIL and the memory are not shared with the request thread that serves heartbeats); or give heartbeats a longer lock timeout plus server-side queuing so a slow query delays rather than drops them.
- The Rust server presumably does not have the GIL problem, but the unbounded-materialization and no-cancellation points may apply there too; worth checking.
Related
- Improving performance when performing summary over a day and over a time period #49 (query performance over longer periods)
- Year / All time: month-sized category requests, timeouts error out with no progress, aborted requests keep running on the server aw-webui#1094 (Year / All time view behaviour)
- macOS (Swift): heartbeats are unserialized fire-and-forget tasks; pile up in parallel when the server is slow, failures are dropped aw-watcher-window#160 (macOS watcher parallel heartbeats under server slowness)
Metadata
Metadata
Assignees
Labels
No labels
Activity
TimeToBuildBob commentedon Nov 17, 2026
Reproduced on a synthetic DB (no real data), and opened #226 as a fix for the outage part.
Reproduction on master
277609e: 310k events over 365 days. Three 1 Hz heartbeat senders and three concurrent Year Activity-style queries with a 30 s client timeout. The server ran undermemory.high=400Mto stand in for the swapping:503 Server busybetween 28 s and 205 s. A second run gave 30 slow heartbeats and two 503s.#226: adds a
query_max_eventslimit per timeperiod (default 1M, about 1.3 GB). Before eachquery_bucketfetch it counts the matching rows. An over-limit timeperiod gets a 400 that tells the client to split the range. Same scenario after the fix: 400 in 0.1 s, 52 MB RSS, 0 slow heartbeats, 0 503s. The count adds no measurable cost under the limit. A 1.86M-event "All time" query gets a 400 in 0.57 s with the default limit.Overlap with #173: none on this path. #173 streams
/exportand speeds up heartbeat writes, butquery2still materializes everything, so it doesn't change any of the three causes. It is also currently conflicting, so I kept this one separate and small.Still open from the issue: cancellation on disconnect, slow-query logging, and streaming or lighter
Events in aw-core.fix(query): cap events one query timeperiod may load (query_max_event…
TimeToBuildBob commentedon Nov 17, 2026
#226 (
query_max_events) merged. I checked master869eccfagainst a live server (testing profile,query_max_events = 50, a bucket with 120 events):QueryExceptionin well under a second, with a message pointing at shorter timeperiods / the config keyThat covers the first and third items in the title. A query can no longer load an unbounded result, and in the reproduction above the 503 window came from that memory blow-up.
I'm leaving this open for the second item: no cancellation after client timeout. A query under the limit still runs to completion after the web UI gives up, and with the default limit of 1M events (about 1.3 GB) that is still wasted work. The fix would be a server-side deadline matching the client's timeout, checked between query2 statements. If the cap is enough for you, close this.