Repository navigation
Conversation
| DataSourceExec | ||
| ``` | ||
|
|
||
| This plan: |
There was a problem hiding this comment.
can you please expand here with an explanation of how this operator works? This is not enough detail to follow I don't think
There was a problem hiding this comment.
fixed - the walkthrough now goes stage by stage: SortExec does a blocking, spill-capable sort of all N rows because the window needs each partition contiguous and ordered; BoundedWindowAggExec walks the sorted stream and numbers every row (it holds state only for the open partition, but touches all N); the filter then keeps rn <= K and throws the rest away.
Also folded in the concrete ratio (10M rows / 1K partitions / K=3 → 3K survivors, 0.03%), and moved "How It Works" before the usage section so the new operator is explained before it appears in the second EXPLAIN.
| operator works, and reports the measured performance — including the cases | ||
| where it is *not* a good fit. | ||
|
|
||
| This post grew out of a draft section [@alamb] wrote for the [DataFusion |
| SET datafusion.optimizer.enable_window_topn = false; | ||
| ``` | ||
|
|
||
| ## How It Works |
There was a problem hiding this comment.
I would put this into beginning
| | --------------------------------- | ------: | -------: | ----------- | | ||
| | 100 partitions (100K rows each) | 43 ms | 174 ms | **4.0x** | | ||
| | 1K partitions (10K rows each) | 71 ms | 146 ms | **2.1x** | | ||
| | 10K partitions (1K rows each) | 619 ms | 128 ms | **0.2x** (regression) | |
There was a problem hiding this comment.
would be good to come up with golden partition num number and also the reason why regression happens
There was a problem hiding this comment.
Measured this on a released build (55.2.0, exact h2o group-by Q8 shape, 10M rows, K=2, min of 4 runs) — two things turned up:
- The published table predates #23096 (shared encoder/reservation), which is already in the 55.0.0 changelog — so the 10K-partition regression doesn't reproduce on released 55.x: the rewrite is ~5.6x faster there. The old numbers are real for the #21479-era build, so the post keeps them labeled "as measured in #21479" and adds a fresh sweep.
- On 55.2.0 the crossover sits between ~50K and ~60K partitions — roughly 150–200 rows per partition at K=2. The reason the regression happens beyond that: the enabled-plan cost grows linearly with P (~5–6 µs per distinct partition key — a hash probe into P heap states per row, one heap + hash-map entry per key, an O(P log P) emit sort), while the sort it replaces stays roughly flat in P. And unlike SortExec, PartitionedTopKExec has no spill path (memory is O(K×P×row_size)), so it can't degrade gracefully the way the sort does. The window operator on top adds its own ~4 µs/partition on top of that.
Reworking the section now with the fresh table and fixing the "follow-up work on main" sentence.
There was a problem hiding this comment.
I don't think we need to try and detail the history of the development -- i recommend dropping this section entirely and just presenting the numbers in the next section "Performance Improvement"
What: restructure and expand the Top N per group post per review remarks. Why: alamb asked for a fuller explanation of the default plan (L105) and flagged the draft-provenance paragraph as irrelevant detail (L56); comphead asked to move the mechanism section before the usage section (L166). How: expanded the three-stage default-plan walkthrough with per-stage mechanics (blocking spill-capable sort, bounded window numbering, filter selectivity); deleted the provenance paragraph and its now-unused link def; moved How It Works (with its subsections) directly after the default-plan section so the operator is explained before it appears in the second EXPLAIN output.
What: rework the benchmark section — keep the original table labeled as pre-#23096, add a fresh partition sweep on DataFusion 55.2.0, and correct the follow-up-work sentence. Why: comphead asked for a golden partition number and the reason for the regression; on released 55.x the 10K-partition regression does not reproduce because #23096 is already in the 55.0.0 changelog, so the old 'follow-up work on the main branch' claim was wrong. How: new sweep table (h2o group-by Q8 shape, 10M rows, K=2, min of 4 runs, crossover between ~50K and ~60K partitions ≈ 150-200 rows/partition), the two regression mechanisms (linear per-partition overhead vs flat sort baseline; no spill path in PartitionedTopKExec + window-operator per-partition cost), one-line trade-off summary.
|
Thanks @Inf1mum appreciate if you can provide a rendered version link, so we can go though it, please check earlier blogposts how to gen the link |
What: rewrite the prose in a plain first-person voice, cut redundant passages, and keep all technical facts, config names, code blocks, tables, and the heading structure unchanged. Why: alamb asked for the post to be reworked and written in the author's own words rather than as an LLM-generated post. How: deleted the roadmap-style intro paragraph and the pattern-section bullet list that repeated the intro examples; merged the duplicated trade-off paragraphs in the benchmarks section into one; folded the 'if you try this' paragraph into the Get Involved list; replaced formulaic connective phrasing with shorter sentences; wrote the benchmark-intro sentence in first person (I re-ran the same shape on 55.2.0). Diff: 62 insertions, 79 deletions in one file. Signed-off-by: Inf1mum <vadimlazuk5@gmail.com>
|
Both items, done in 5bcf8cd:
Ready for re-review. |
|
thanks, rendered version is much easier to read and follow |
|
Thanks @comphead — glad it reads better now. Left the prose in the same register for the remaining sections; happy to adjust anything else that stands out. |
alamb
left a comment
There was a problem hiding this comment.
Thank you @Inf1mum -- I wrote up quite a bit of feedback, hopefully it is helpful. It will likely take some time to implement but I think this blog will be much better for it
cc @SubhamSinghal as you were the primary author so I suspect you may be interested in this post too
|
|
||
| *[Vadim Lazuk](https://github.com/Inf1mum)* | ||
|
|
||
| "Find the top N rows within each group" is one of the most common shapes in |
There was a problem hiding this comment.
in the intro of the article, you have only a few words to "hook" the reader so I think we should make this as concise as possible. For example, do we need to have all three examples? I think one makes the point
There was a problem hiding this comment.
Also I am not sure I would call it "the most common shapes" -- maybe a "common shape" ?
| to compute the window function, and then throwing almost all of the sorted | ||
| output away. DataFusion 55 recognizes the pattern: with | ||
| [datafusion.optimizer.enable_window_topn] enabled, the full sort is replaced | ||
| by a new `PartitionedTopKExec` operator that keeps only the top N rows per |
There was a problem hiding this comment.
maybe link to the operator documentation here?
| ) WHERE rn <= 5; | ||
| ``` | ||
|
|
||
| Until recently, DataFusion evaluated this query by sorting the *entire* input |
There was a problem hiding this comment.
I think you could make this paragraph a more concise (it is the intro) by removing the detail on how to enable and the operator name - the main point of this paragraph I think is that DataFusion now has a specialized operator for this kind fo query and that it goes XXX times faster (we should figure out what XXX is and include that).
|
|
||
| The query above computes the top five rows per `category`. The same shape | ||
| covers the latest events per device, the highest-paid employees per | ||
| department, and many other screens and reports. It is also query 8 of the |
There was a problem hiding this comment.
What does "screen" mean in this context? I don't think the query 8 / db-benchmark reference is super releavant here and can be cut . The fact that the query is in benchmarks shouldn't be a surprise given the post claims it is a common query pattern
There was a problem hiding this comment.
(and to be clear, I am asking what screen means because I think you could remove it and make this more concise)
| [h2o group-by benchmark]: https://github.com/h2oai/db-benchmark | ||
| [#13586]: https://github.com/apache/datafusion/issues/13586 | ||
|
|
||
| What matters for the optimizer is the ratio: the filter keeps a *small, fixed |
There was a problem hiding this comment.
THis paragraph feels out of place / too early -- you haven't introduced sorting or sort based plans yet and this is some sort of analysis of that plan
| | 10,000 | 1,000 | 159 ms | 885 ms | 0.18x (5.6x faster) | | ||
| | 20,000 | 500 | 205 ms | 791 ms | 0.26x (3.9x faster) | | ||
| | 50,000 | 200 | 416 ms | 500 ms | 0.83x (≈ parity) | | ||
| | 60,000 | ~167 | 454 ms | 289 ms | 1.57x (regression) | |
There was a problem hiding this comment.
I would personally summarize this as "above 50,000 partitions performance gets worse"
Although I thought this was fixed in datafusion 56 ?
| per partition** at `K = 2`. Past that the rewrite loses ground for two | ||
| reasons: | ||
|
|
||
| - **Per-partition overhead grows linearly with P.** Every row probes the |
There was a problem hiding this comment.
Why is this analysis and below important to the story (aka that this is a good operator)?
|
|
||
| ## Status and Roadmap | ||
|
|
||
| As of DataFusion 55.0.0: |
There was a problem hiding this comment.
List should be updated to be "as of 56"
| `datafusion.optimizer.enable_window_topn`. | ||
| - `DENSE_RANK` support has landed on the main branch | ||
| ([#23869]) and is expected in an upcoming release. | ||
| - The optimization is **not yet enabled by default**. Work to flip the |
There was a problem hiding this comment.
this ticket is now fixed -- so the optimization will be enabled in 57 by default.
|
|
||
| - [Further "per partition" top-k optimizations][#24060] — the umbrella for | ||
| the next round of improvements. | ||
| - [Support filters with embedded projections][#21596] — widen the set of |
There was a problem hiding this comment.
I think just the umbrella item (which links to the other tickets) is adequate
Rendered version: https://inf1mum.github.io/datafusion-site/blog/2026/11/13/topk-per-partition/ — GitHub Pages preview built from this branch with the ASF-Pelican build (same approach as the preview in #203).
Add a self-contained blog post about the "TopK per partition" optimization for window functions (
PartitionedTopKExec/datafusion.optimizer.enable_window_topn), as requested in apache/datafusion#27233: apache/datafusion#27233About the post
The post expands the draft section written for the DataFusion 55 release post (the draft in the issue, by Andrew Lamb, who also wrote the request) into a full informative post following the "low-key DataFusion pitch" pattern of the Dynamic Filters and Sort Pushdown posts. Structure:
SET datafusion.optimizer.enable_window_topn = true, before/afterEXPLAINoutput, which queries theWindowTopNrule applies to, and how to turn it offRowConverter/MemoryReservation, ROW_NUMBER vs RANK retention semantics, emission orderROW_NUMBER+RANKin 55,DENSE_RANKon main (#23869), the open enable-by-default EPIC (#24608 / #27197) — stated accurately as of todayCoordination / credits
Per the issue's note that blog posts help raise author visibility: the post credits SubhamSinghal (feature implementation), 2010YOUY01 and kosiew (reviews), jayzhan211 (memory optimizations), saadtajwar, stuhood, viirya, and zhuqi-lucas (further contributions), and Andrew Lamb (draft section this post grew out of), with links to their GitHub profiles.
Facts and validation
datafusion/physical-optimizer/src/window_topn.rs,datafusion/physical-plan/src/sorts/partitioned_topk.rs,config.rs), thewindow_topn.sltsqllogictest EXPLAIN output, and the linked PRs — no invented numbers. The "enable by default" work is described as in-flight (the PR is still open at the time of writing).2026-07-20-sort-pushdown.md,2026-10-24-point-queries.md).typoscheck passes. The front matter is identical in shape to sibling posts, and CI's pelican build will be the authoritative check.Review
Review would be most valuable from the feature authors, in particular SubhamSinghal (https://github.com/SubhamSinghal), jayzhan211 (https://github.com/jayzhan211), and alamb (https://github.com/alamb), to check that the technical description and benchmark framing match their intent.