Repository navigation
fix(peewee): retain accepted long events in overlap queries - #197
TimeToBuildBob wants to merge 5 commits into
Conversation
[Medium impact] The PR appears safe to merge under its documented single-writer assumption. SummaryKeeps long events in bounded reads and counts while preserving an indexed timestamp range.
Reviews (4) · Last reviewed commit: "fix(peewee): raise the duration bound be..." · Reviewed by Greptile |
🤖 AI code reviewIntroduces a per-bucket dynamic lookback for range queries in the peewee storage layer: replaces the fixed 24h prefilter with a bound computed from the bucket's longest stored duration, combined with an in-memory floor raised before writes. Adds a reentrant lock to serialize duration-bound computation, writes, and reads, and adds a new test file covering long-event overlap and concurrency scenarios. Safe to merge — no P0/P1 findingsConfidence 5/5 ✅ No findings. The diff looks correct to me on this pass. Files changed (2) — the diff as I read it
Previous review passes
Reviewed Maintainer commands
|
Git-Session-Id: d8a22e8f-4386-4789-bab9-5ddd743d4b08
Git-Session-Id: d8a22e8f-4386-4789-bab9-5ddd743d4b08
02f5c80 to
7494660
Compare
|
Rebased onto current master ( Verification after rebase: 395 tests passed, 5 skipped (suite excluding GitHub now reports the branch mergeable. New-head CI is running; no merge performed. |
This comment has been minimized.
This comment has been minimized.
|
CI-green and mergeable (Greptile 5/5) — waiting only on a maintainer click. This PR is ready to merge, but the bot has pull-only access to this repo and can't self-merge — surfacing it here so it isn't lost. The monitoring loop will stop re-flagging it now that this note is posted. |
Removing the 24h prefilter made every range query scan the bucket's whole history. Restore an indexable lower bound from the bucket's longest stored duration: computed lazily with MAX(duration), raised on insert, insert_many, replace and replace_last, and reset when buckets are created or deleted. A delete can leave it high, which only widens the scan. 3M-event synthetic bucket, one-hour range, count: master (24h) 0.77ms, no bound 1288ms, bounded 0.17ms (30s max) / 1.28ms (one 48h event). Git-Session-Id: ee0ba498-ccac-4974-9a3f-5cdd56c11532
This comment has been minimized.
This comment has been minimized.
A write now raises a per-instance floor before it reaches the database, and the lookback uses max(database maximum, floor). A reader that fills the cache from an older maximum, or a batch that fails after some chunks were saved, can no longer leave the bound below a stored duration. Bounds are reset per bucket on create/delete instead of globally. Git-Session-Id: ee0ba498-ccac-4974-9a3f-5cdd56c11532
|
@greptileai review |
Hold an instance-wide reentrant lock from bound computation through query consumption, and across duration-changing writes and bucket resets. Raising the floor alone did not protect an already-built WHERE clause. Keep the indexed range and existing schema; serialize instead of adding a per-query MAX or migration. Add eight reader/writer interleaving regressions that fail before the lock. Git-Session-Id: 0ed84369-5333-52c9-a08f-f4252338f90b
Review adjudication — fix verified locally; await current-head checks
Recommendation: Hold merge until current-head CI/review finishes; then maintainer judgment on the serialization trade-off. No merge performed. |
Summary
Fixes #194. The
timestamp >= start - 24hprefilter silently dropped accepted events longer than 24h from bounded reads and counts. The lookback is now bounded by the bucket's longest stored duration instead of a fixed 24h, per review: correct for any accepted duration, and still an indexable range oneventmodel_bucket_id_timestamp._bucket_max_durationis computed lazily per bucket withMAX(duration)(one bucket scan per process, ~200 ms at 3M events). It is cached in memory.insert_one,insert_many,replaceandreplace_lastraise a per-instance floor before writing, and the bound ismax(database maximum, floor). An instance-wide reentrant lock spans bound computation through query consumption, all duration-increasing writes, and bucket resets, preventing a write from committing between lookback selection and query execution. A partially failed batch can only leave the floor too high. These operations serialize across buckets; multi-process writers remain unsupported. Bucket create/delete resets it per bucket. A delete can leave it high, which only widens the scan and stays correct.bucket_keys, the cache assumes this process is the bucket's writer. A second process inserting a longer event would not raise it until restart.Reproduction and verification
On master
c7821899, insert a 48-hour event at2026-10-01T00:00:00Zand query2026-10-02T12:00Z–13:00Z: the unbounded read contains it, but the bounded read is empty and the count is zero.tests/test_peewee_long_events.py: 24/48/240-hour events; interior and inclusive boundary hits; non-overlaps; one-sided/unbounded/empty ranges; equivalent timezone offsets; limits/order; unchanged stored content. New tests check that a later, longer event widens the bound through insert, insert_many, replace and replace_last; that a write racing a cold read, or a batch failing partway, still keeps its events; and that recreating a bucket resets the bound. The query-plan test now asserts a two-sidedtimestamp>? AND timestamp<?index range. 52 passed. Eight additional regressions interleave all four write APIs between range construction and execution for both reads and counts; all eight failed before the lock and pass afterward.tests/test_flood.py: 408 passed, 5 skipped.make typecheck: 37 files clean. Pinned Ruff clean.Peewee already clips returned copies, including
query_bucketAQL results, so an interior result is the clipped hour, not the whole 48 hours. This change affects inclusion only.Benchmark: 3,000,000-event bucket (before the serialization fix)
Disposable synthetic SQLite DB: one bucket, 3M ordinary 30-second events, one per minute (~5.7 years). One-hour ranges near the end and in the middle of history; warm, median of 7. Every variant returns the same 61 rows.
The bounded fix is as fast as master's lossy prefilter: faster when the bucket holds no long events, and about master's cost at a 48h bound. The trade-off is that a single very long event widens every range query on its bucket. One cold
MAX(duration)per bucket per process costs 203 ms at 3M rows. Synthetic warm measurements, not a production latency claim. These measurements predate the serialization fix in6b61eeb; concurrent throughput under the lock has not been benchmarked.