Repository navigation
perf: stop computing an HLL estimate on every merge and size() call - #7629
Conversation
`SparkHllSketch` and `SparkHllUnion` inferred their heap size from `estimate()`, and `SparkHllUnion::merge` called `estimate()` on both sides before every update to keep a `dense` flag for that inference. `datasketches` 0.6.0 exposes the size directly: `estimated_size()` reads the coupon or register array length in O(1) and is exact. Use it, and drop the `layout` inference, both `dense` flags and the `preamble` module. On 0.6.0 this makes Final-mode `hll_sketch_agg` and `hll_union_agg` over many tiny groups about 20% faster (211 to 169 ns per partial row at lgConfigK 18). It matters more once Comet moves to a datasketches release with apache/datasketches-rust#317, which defers the register cache rebuild after a dense merge until the next query. The per-merge `estimate()` forced that rebuild on every merge. Built against apache#317, merging an already dense sketch drops from 266 us to 5.5 us, close to the 5.2 us a plain register-wise max takes. Part of apache#7341.
ef10fc2 to
ffa0694
Compare
andygrove
left a comment
There was a problem hiding this comment.
Thanks @comphead. Using the crate's estimated_size() is a nice simplification, and the exact-match assertion in heap_size_follows_the_crates_layout is a stronger test than the old bound. I checked 0.6.0's implementation. It reads the boxed coupon slice or the register array length, and the Estimator holds no heap, so nothing is missed.
I ran the hll_agg bench on x86 against the merge base, alternating over three rounds. hll_sketch_agg drops from about 1.95 ms to 1.71 ms. hll_union_agg and hll_sketch_agg_final_dense are unchanged within noise, which matches your note that the dense path waits on the next datasketches release with apache/datasketches-rust#317. LGTM.
viirya
left a comment
There was a problem hiding this comment.
Nice cleanup. Two small follow-ups inline, both fine to fold into the PR that bumps datasketches to a release with apache/datasketches-rust#317.
| /// Merge a sketch into the union without querying it. Querying after each dense merge would | ||
| /// rebuild the register cache, which apache/datasketches-rust#317 defers until it is needed. |
There was a problem hiding this comment.
On 0.6.0 the rebuild happens inside update (merge_array_same_lgk calls rebuild_estimator_from_registers), and estimate() only reads the cache, so this reads as if querying is what triggers it today. Maybe:
/// Merge a sketch into the union without querying it. With apache/datasketches-rust#317 the
/// register cache rebuild after a dense merge waits for the next query, and an `estimate()`
/// here would force it on every merge.
| /// register array promotes it however little that sketch has seen, so the charge has to follow | ||
| /// the gadget's mode, not its estimate. |
There was a problem hiding this comment.
The hand-edited preamble below existed to defeat the estimate-based mode inference, which is gone. Since heap_size() now reads the container, merging any dense sketch and asserting 1 << 16 covers the same promotion, without tying the test to the HLL preamble offsets. The "not its estimate" here can go with it.
Which issue does this PR close?
Part of #7341. #7341 stays open for item 2 (a
GroupsAccumulator) and for thedatasketchesbump once a release includes apache/datasketches-rust#317.Rationale for this change
#7345 fixed the problem in the title of #7341. Item 3 is still real on
main: merging a partial sketch that is already dense costs about 270 µs at lgConfigK 18, while the register-wise max it needs takes 5 µs.datasketches0.6.0 rebuilds its register cache, onef64division per register, after every dense merge.apache/datasketches-rust#317 (merged 2026-11-04, not released yet) defers that rebuild until the next query. Comet would not benefit from it.
SparkHllUnion::mergecallsestimate()on the union and on the incoming sketch before every update, to keep adenseflag for the heap-size inference, andheap_size()callsestimate()on every accumulatorsize(). Each of those calls would force the deferred rebuild.datasketches0.6.0 exposes the size directly.HllSketch::estimated_size()andHllUnion::estimated_size()read the coupon array or register array length in O(1), and the result is exact. The estimate-based inference predates that API, since 0.3.0 kept the layout private.What changes are included in this PR?
SparkHllSketch::heap_size()andSparkHllUnion::heap_size()return the crate'sestimated_size().layoutmodule, bothdenseflags and thepreamblemodule are removed, somergeandsize()no longer compute estimates.heap_size_follows_the_crates_layoutnow asserts thatheap_size()equals the allocation at every step, instead of "never below, at most one size step above".How are these changes tested?
The existing heap-size tests cover it.
heap_size_follows_the_crates_layoutnow requires an exact match with the allocation the crate serializes at every cardinality, through each SET resize and the promotion.high_lg_config_k_is_charged_for_what_it_holds,union_heap_size_follows_the_crates_layoutandsingleton_groups_at_high_lg_config_k_fit_a_small_poolare unchanged. Sketch results do not change: only the size accounting is touched, and the probe below gave identical output bytes in every build.The numbers come from a throwaway example (not in this PR) that runs a Final
AggregateExecover partial states at lgConfigK 18, with rows interleaved one partial per group per batch, the way a reducer reads map outputs. Release build on an Apple M-series laptop,hll_sketch_agg, ns per partial row, median of 5 runs over two interleaved rounds.hll_union_agggives the same picture.With
datasketches0.6.0, as shipped by this PR:mainWith
datasketchesat apache/datasketches-rust@5f080f98e5 (#317, still version 0.6.0, through[patch.crates-io], not part of this PR):mainSparkHllUnion::mergeof a dense sketchA plain 2^18 register-wise max takes 5.2 µs, so with #317 a dense merge costs about what the merge itself has to do. Differences under 5% are within run-to-run noise here.