Skip to content

fix(peewee): retain accepted long events in overlap queries - #197

Open
TimeToBuildBob wants to merge 5 commits into
ActivityWatch:masterfrom
TimeToBuildBob:fix/long-event-overlap
Open

TimeToBuildBob wants to merge 5 commits into
ActivityWatch:masterfrom
TimeToBuildBob:fix/long-event-overlap

Conversation

@TimeToBuildBob

@TimeToBuildBob TimeToBuildBob commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #194. The timestamp >= start - 24h prefilter silently dropped accepted events longer than 24h from bounded reads and counts. The lookback is now bounded by the bucket's longest stored duration instead of a fixed 24h, per review: correct for any accepted duration, and still an indexable range on eventmodel_bucket_id_timestamp.

  • _bucket_max_duration is computed lazily per bucket with MAX(duration) (one bucket scan per process, ~200 ms at 3M events). It is cached in memory.
  • insert_one, insert_many, replace and replace_last raise a per-instance floor before writing, and the bound is max(database maximum, floor). An instance-wide reentrant lock spans bound computation through query consumption, all duration-increasing writes, and bucket resets, preventing a write from committing between lookback selection and query execution. A partially failed batch can only leave the floor too high. These operations serialize across buckets; multi-process writers remain unsupported. Bucket create/delete resets it per bucket. A delete can leave it high, which only widens the scan and stays correct.
  • One second of slack absorbs decimal/float rounding. An overflowing lookback means no lower bound.
  • No schema or index migration, no duration cap, no stored-history rewrite. UTC normalization, the inclusive end-overlap predicate, clipping, sorting and limits are unchanged.
  • Like bucket_keys, the cache assumes this process is the bucket's writer. A second process inserting a longer event would not raise it until restart.

Reproduction and verification

On master c7821899, insert a 48-hour event at 2026-10-01T00:00:00Z and query 2026-10-02T12:00Z–13:00Z: the unbounded read contains it, but the bounded read is empty and the count is zero.

  • tests/test_peewee_long_events.py: 24/48/240-hour events; interior and inclusive boundary hits; non-overlaps; one-sided/unbounded/empty ranges; equivalent timezone offsets; limits/order; unchanged stored content. New tests check that a later, longer event widens the bound through insert, insert_many, replace and replace_last; that a write racing a cold read, or a batch failing partway, still keeps its events; and that recreating a bucket resets the bound. The query-plan test now asserts a two-sided timestamp>? AND timestamp<? index range. 52 passed. Eight additional regressions interleave all four write APIs between range construction and execution for both reads and counts; all eight failed before the lock and pass afterward.
  • Core suite excluding tests/test_flood.py: 408 passed, 5 skipped. make typecheck: 37 files clean. Pinned Ruff clean.
  • Cross-repo HTTP regressions (bounded GET/count, CSV, AQL) are tracked in test(server): guard long-event overlap consumers aw-server#215 / Guard bounded REST, CSV and AQL reads against long-event omission aw-server#213.

Peewee already clips returned copies, including query_bucket AQL results, so an interior result is the clipped hour, not the whole 48 hours. This change affects inclusion only.

Benchmark: 3,000,000-event bucket (before the serialization fix)

Disposable synthetic SQLite DB: one bucket, 3M ordinary 30-second events, one per minute (~5.7 years). One-hour ranges near the end and in the middle of history; warm, median of 7. Every variant returns the same 61 rows.

Variant Count, recent 1h Count, middle 1h Read limit 100, recent 1h
master (24h prefilter, lossy) 0.77 ms 0.75 ms 2.03 ms
previous head (no lower bound) 1288 ms 605 ms 1508 ms
this head, max duration 30 s 0.17 ms 0.17 ms 1.22 ms
this head, after one 48 h event 1.28 ms 1.27 ms 2.73 ms

The bounded fix is as fast as master's lossy prefilter: faster when the bucket holds no long events, and about master's cost at a 48h bound. The trade-off is that a single very long event widens every range query on its bucket. One cold MAX(duration) per bucket per process costs 203 ms at 3M rows. Synthetic warm measurements, not a production latency claim. These measurements predate the serialization fix in 6b61eeb; concurrent throughput under the lock has not been benchmarked.

@greptile-apps

greptile-apps Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge under its documented single-writer assumption.

Summary

Keeps long events in bounded reads and counts while preserving an indexed timestamp range.

  • Peewee range reads and counts use each bucket’s longest event duration.

Reviews (4) · Last reviewed commit: "fix(peewee): raise the duration bound be..." · Reviewed by Greptile

Comment thread tests/test_peewee_long_events.py Resolved
@TimeToBuildBob

TimeToBuildBob commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor Author

🤖 AI code review

Introduces a per-bucket dynamic lookback for range queries in the peewee storage layer: replaces the fixed 24h prefilter with a bound computed from the bucket's longest stored duration, combined with an in-memory floor raised before writes. Adds a reentrant lock to serialize duration-bound computation, writes, and reads, and adds a new test file covering long-event overlap and concurrency scenarios.

Safe to merge — no P0/P1 findings

Confidence 5/5

✅ No findings. The diff looks correct to me on this pass.

Files changed (2) — the diff as I read it
  • aw_datastore/storages/peewee.py — Adds RLock and duration-bound tracking, decorates writes/reads with the lock, and changes _where_range to use a dynamic max-duration lookback.
  • tests/test_peewee_long_events.py — Adds tests for long-event overlap, ordering, limits, lookback widening, bucket reset, and write/read concurrency.
Previous review passes
commit score findings engine when
02f5c80426eb 5/5 0 subscription 2026-10-28 23:22 UTC
74946601dbc7 5/5 0 subscription 2026-10-29 15:24 UTC
c2546c44f418 3/5 1 llm 2026-11-17 14:52 UTC

Reviewed 6b61eeb4e1b3 · openrouter/deepseek/deepseek-v4-flash-0731 · llm engine · 99s · about this reviewer

Maintainer commands

@TimeToBuildBob review (own line) — fresh review · @TimeToBuildBob fix — a worker acts on the findings. Once per comment; 👀 = received.

Git-Session-Id: d8a22e8f-4386-4789-bab9-5ddd743d4b08
Git-Session-Id: d8a22e8f-4386-4789-bab9-5ddd743d4b08
@TimeToBuildBob

Copy link
Copy Markdown
Contributor Author

Rebased onto current master (1dcab4c) and pushed 7494660. The only conflict was the Linux runner pin: kept master's Ubuntu 22.04 configuration and dropped this branch's redundant Ubuntu 24.04 CI commit. Both product/test commits are unchanged (git range-diff confirms).

Verification after rebase: 395 tests passed, 5 skipped (suite excluding tests/test_flood.py, isolated temporary data/config/cache directories); mypy clean on 37 source files; pinned Ruff and pre-commit hooks passed. The first non-isolated run hit an existing random test-bucket ID collision; the isolated rerun passed without source changes.

GitHub now reports the branch mergeable. New-head CI is running; no merge performed.

@TimeToBuildBob

This comment has been minimized.

@TimeToBuildBob

Copy link
Copy Markdown
Contributor Author

CI-green and mergeable (Greptile 5/5) — waiting only on a maintainer click.

This PR is ready to merge, but the bot has pull-only access to this repo and can't self-merge — surfacing it here so it isn't lost. The monitoring loop will stop re-flagging it now that this note is posted.

Removing the 24h prefilter made every range query scan the bucket's whole
history. Restore an indexable lower bound from the bucket's longest stored
duration: computed lazily with MAX(duration), raised on insert, insert_many,
replace and replace_last, and reset when buckets are created or deleted. A
delete can leave it high, which only widens the scan.

3M-event synthetic bucket, one-hour range, count: master (24h) 0.77ms,
no bound 1288ms, bounded 0.17ms (30s max) / 1.28ms (one 48h event).

Git-Session-Id: ee0ba498-ccac-4974-9a3f-5cdd56c11532
@TimeToBuildBob

This comment has been minimized.

Comment thread aw_datastore/storages/peewee.py Resolved
Comment thread aw_datastore/storages/peewee.py Resolved
Comment thread aw_datastore/storages/peewee.py Resolved
A write now raises a per-instance floor before it reaches the database,
and the lookback uses max(database maximum, floor). A reader that fills
the cache from an older maximum, or a batch that fails after some chunks
were saved, can no longer leave the bound below a stored duration.
Bounds are reset per bucket on create/delete instead of globally.

Git-Session-Id: ee0ba498-ccac-4974-9a3f-5cdd56c11532
@TimeToBuildBob

Copy link
Copy Markdown
Contributor Author

@greptileai review

Comment thread aw_datastore/storages/peewee.py Resolved
Hold an instance-wide reentrant lock from bound computation through query consumption, and across duration-changing writes and bucket resets. Raising the floor alone did not protect an already-built WHERE clause. Keep the indexed range and existing schema; serialize instead of adding a per-query MAX or migration. Add eight reader/writer interleaving regressions that fail before the lock.

Git-Session-Id: 0ed84369-5333-52c9-a08f-f4252338f90b
@TimeToBuildBob

Copy link
Copy Markdown
Contributor Author

Review adjudication — fix verified locally; await current-head checks

  • Fixed: The remaining AI-review P1 (640b2dea4b85) was a real blocking race. Pushed 6b61eeb4e1b34dcfb311f422979a2b7ef068c942: an instance-wide RLock covers lookback computation through query consumption, all duration-increasing writes, and bucket resets. Replied on the finding's inline thread and verified its resolution. Eight new read/count × write-path regressions failed before the fix and pass afterward.
  • Remaining: No unresolved reviewer threads. No new finding was dismissed. The earlier external-writer finding remains an explicitly accepted limitation: separate storage instances/processes are outside this cache's single-instance writer contract (Greptile withdrew it on its inline thread). The lock serializes reads and writes across buckets; that throughput trade-off needs maintainer acceptance.
  • Verified against: Both published reviewers covered c2546c44f41805e246168341923e9b8b3e39e548 at session start. The bug classification was reproduced there; the fix and tests were checked on new head 6b61eeb4e1b34dcfb311f422979a2b7ef068c942. Read-only independent inspection found no blocking concerns. The prior Greptile 5/5 is not a review of this new head.
  • CI: New-head Linux/macOS/Windows tests, lint and CodeQL analysis are pending. Local verification: 408 passed, 5 skipped, excluding tests/test_flood.py; 54 Peewee tests passed; mypy clean on 37 files; pinned Ruff and commit hooks passed. Updated the PR description to distinguish earlier latency measurements from the unbenchmarked lock contention.
  • Domain risk: Please exercise a bulk import while heartbeats and bounded GET/count queries run, checking responsiveness across buckets. The indexed query plan is preserved, but lock-contention throughput has not been benchmarked. No schema migration or multi-process writer support was added.
  • Convergence: Detector reports round_convergence.stable_rounds = 0 (required: 2), with no unresolved Greptile P1/P2 threads. That is a historical round counter, not evidence of two stable current-head passes. No additional Greptile review was triggered.

Recommendation: Hold merge until current-head CI/review finishes; then maintainer judgment on the serialization trade-off. No merge performed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Peewee range queries omit accepted events longer than 24 hours

1 participant