feat(clickhouse): make rollup routing reach real windows; add backfillDaily and latest-value gauge projections - #39
Merged
levivannoort merged 3 commits intoSep 4, 2026
Conversation
…ackfillDaily getTotalBatch() always scanned the raw events table, even for windows the daily rollup answers — for Appwrite billing that is a whole-month raw scan per metric batch per project per hour, and it is the shape that saturates ClickHouse once the raw table grows. Batched event totals now route through the same source selection sum()/getTotal() use: closed-day windows read the daily rollup, open windows read rollup + today's raw tail, and anything the rollup cannot answer (non-rollup filter columns, cursors, sub-day bounds) stays raw. Decisions land in the route log under `getTotalBatch`, and the dualReadSampleRate canary gets a batched twin that logs per-metric drift. Gauges are untouched — they have no rollup. Routing is only correct once the rollup covers the window, and the daily MV was created without POPULATE, so backfillDaily($from, $to) runs the exact MV aggregation over a half-open, UTC-midnight-aligned window. SummingMergeTree folds duplicates by adding them, so the call refuses misaligned bounds and refuses a window that already holds rollup rows unless force is passed after the caller cleared the range.
Greptile SummaryThe PR routes batched ClickHouse event totals through daily, hybrid, or split aggregate sources; adds an operator-managed daily-rollup backfill API; and introduces latest-value gauge projections.
Confidence Score: 5/5The PR appears safe to merge; no blocking or independently actionable new defect remains. No accepted new findings remain, and all previous findings were resolved: the rollup-coverage and concurrency concerns were withdrawn after their operational contracts were clarified, while the timezone-offset issue is fixed by UTC normalization before alignment validation. Important Files Changed
Reviews (3): Last reviewed commit: "feat(clickhouse): split route for non-al..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Contributor
Author
|
Production coverage data (fra1, the largest deployment) that changes the rollout cost of this PR: The daily MV has been live since provisioning — a per-month parity check of Implications:
Also relevant for reviewers: the divergence mechanism (rollup can drift below raw after tenant purges) is a good argument for keeping the |
…rrency contract
A bound carrying its own offset ('2026-01-01 00:00:00+05:00') passed the
midnight check in its own zone but was bound offset-less, so ClickHouse
read it as UTC midnight — a different instant than the caller intended.
Bounds now normalize to UTC before validation, so offset midnights that
are not UTC midnights are refused. backfillDaily() is also documented as
a single-operator migration primitive: the overlap check and the INSERT
are separate statements, so concurrent calls for the same window can
double the rollup — serialize, and parity-check before routing traffic.
…auge projections Two measured gaps, both verified against the largest production install. 1. Routing never fired for the windows production uses. daily/hybrid require a midnight-aligned start, but billing cycles are anchored at the moment a team upgraded: 0 of 451,293 production teams have a midnight-aligned invoice date. Every billing read therefore fell to raw. The new 'split' route reads the interior whole days from the rollup and only the partial head/tail days from raw, so a window of any shape uses the rollup for everything it can answer. Measured on 12 real high-volume tenants x 5 billing-shaped windows: 60/60 totals identical to raw, 3.71B rows read down to 221M (16.8x). Split also fixes a pre-existing correctness bug it inherited from the daily route: an inclusive midnight upper bound covers the midnight instant, which a day-granularity row cannot express, so translating `<= midnight` to `< midnight` silently dropped rows timestamped exactly at the bound. Those windows now route split, which reads that instant from raw. A non-window time filter (notBetween and friends) carves a shape no day row can honor and now pins the read to raw instead of over-counting. 2. Gauge projections could not collapse a series. Their GROUP BY carried raw `time`, making them 1:1 with the base table — measured in production: 964M projection rows against 966M base rows, i.e. column pruning only. The added latest-value slate is keyed (tenant, metric, dims) with no time key, one argMax state per series: 966M samples collapse to 3.28M series (294 samples per series), and force_optimize_projection now accepts the grouped prefetch shape that it refused before. The windowed slate is unchanged — windowed grouped reads still need the time predicate expressible on the projection — and the new slate's merge cost tracks its own row count, ~0.3% of the base table's. Existing installs must MATERIALIZE the new projections once; setup() only attaches them to parts written afterwards. Production already shows why that step matters: the current slate covers 39-43 of 45 active parts, and a partially covered projection is unusable for the query.
levivannoort
changed the title
loks0n
approved these changes
Sep 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Rollup routing existed but could not fire for the windows production actually issues, and the gauge projections could not collapse a series. Both were measured against the largest production install (fra1: 275 GiB / 7.3 B raw events, 966 M gauge samples), not inferred.
Routing never reached billing.
daily/hybridboth require a midnight-aligned start. Billing cycles are anchored at the moment a team upgraded — 0 of 451,293 production teams have a midnight-aligned invoice date (SELECT SUM(TIME(billingCurrentInvoiceDate)='00:00:00') ...→ 0). So every billing read fell through to a full-cycle raw scan, which is the shape that saturated the box (the top query by CPU: 71 k executions/hour, 5,479 CPU-s, 487 GiB read).Gauge projections were a copy of the base table. Their
GROUP BYcarried rawtime, so they held one row per sample: production shows 964 M projection rows against 966 M base rows — column pruning only, no aggregation benefit.force_optimize_projection = 1on the billing prefetch shape returnedPROJECTION_NOT_USED.What
1.
splitroute — interior whole days from the rollup, only the partial head/tail days from raw, folded by an outerSUMoverUNION ALL. Any window shape now uses the rollup for everything it can answer. Wired into bothsum()/getTotal()andgetTotalBatch().2.
backfillDaily($from, $to, $force)— runs the MV's exact aggregation over a half-open, UTC-midnight-aligned window, because the MV was created withoutPOPULATE. Bounds normalize to UTC before validation (an offset midnight that is not a UTC midnight is refused); refuses a window that already holds rollup rows unlessforce; documented as a single-operator migration primitive (the check and the INSERT are separate statements — ClickHouse has no cross-statement transaction, andSummingMergeTreeadds duplicates).3. Latest-value gauge projections (
p_latest_by_*) — keyed(tenant, metric, dims)with no time key, oneargMaxstate per series. Production: 966 M samples collapse to 3.28 M series (294 samples per series). The existing time-keyed slate is unchanged, because windowed grouped reads need the time predicate expressible on the projection.Two correctness bugs fixed on the way
<= midnightcovers the midnight instant; a day-granularity row cannot express it, and thedailyroute translated<=into<, silently dropping rows timestamped exactly at the bound. Found by differential testing against boundary-dense seeds (off by exactly the midnight row's value). Those windows now routesplit, which reads that instant from raw. Regression test:testInclusiveMidnightUpperBoundExcludesTheEndDayButKeepsTheMidnightInstant— 342 = 300 interior + the 42 row at exactly midnight, never the 9999 mid-end-day row.notBetweencarves a mid-day hole no day row can honor, and the split interior would drop it. Such filters now pin the read toraw(hasIrregularTimeFilter), value-asserted.Evidence
Differential parity on production data — the split route's exact SQL against 12 real high-volume tenants × 5 billing-shaped windows (arbitrary time-of-day anchors), 6 metrics each:
Deterministic parity matrix in CI (
ClickHouseSplitParityTest): 11 window shapes — mid-day both ends, mid-day/midnight mixes, day-aligned, straddling today, inclusive at both alignments, sub-day, exactly-one-interior-day, month-long — each asserted equal to a raw scan on both the flat sum and the batch, and asserted to take the intended route, against seeds at midnight exactly / mid-morning / 23:59:59.999 on every day so any edge off-by-one changes a total. 22/22.Gauge projection shapes (compose ClickHouse, identical data): time-keyed slate 2,880 rows read for the prefetch shape vs 200 for the latest slate (14.4x on 2 M rows; the production collapse factor is 294x).
force_optimize_projectionnow accepts the shape it refused.Full suite 376 tests / 1,686 assertions green; Pint and PHPStan clean. Split and backfill tests verified red against the previous implementation.
Deployment
ALTER TABLE … MATERIALIZE PROJECTION p_latest_by_*) —setup()only attaches them to parts written afterwards. Production shows why this matters: the existing slate covers 39–43 of 45 active parts, and a partially covered projection is unusable for the query.backfillDaily. On fra1, 3 of 4 retained months already match raw exactly; the fourth is oneDROP PARTITION+ backfill.dualReadSampleRateon for a parity window (the batched dual-read in this PR coversgetTotalBatch).GAUGE_LOOKBACK_DAYSbound — see appwrite-labs/cloud#5660: a time predicate cannot be evaluated on a projection with no time key, so the bound would force the prefetch off the new projection (measured 200 rows unbounded vs 21,388 bounded). Materialize first, then drop.Cost — measured, and one number corrected
An earlier revision of this description claimed the new projections' write cost "tracks their own row count, roughly 0.3%". That is right for storage and merge and wrong for inserts, so here are the measurements.
NewPartwork is 87 s/hour (0.024 cores); +37.5% of that is +0.009 coresSo: a large relative insert cost on a tiny absolute base. Worth it, but stated honestly rather than hidden behind the storage ratio.
Also verified while reviewing:
applyFilters()applies only the tenant and filter predicates — noLIMIT/OFFSETleaks into the split branches (which would have silently truncated a branch), matchingsumFromTable()andsumHybridDailyAndRaw().Parity coverage extended
The first production run covered plain metric batches. Billing's network reads also carry
resourceTypefilters (= 'site'/!= 'site'), which route through the rollup's ownresourceTypecolumn — that shape was untested, so it was run too: 6 tenants × 2 filter shapes × 2 windows = 24 comparisons, 0 mismatches.