Repository navigation
perf: combine small input batches before a native sort that merges them #7696
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -47,7 +47,7 @@ use datafusion::{ | |
| }, | ||
| }; | ||
| use datafusion_comet_operators::{ | ||
| CometFilterExec, CometProjectionExec, MembershipKeyReductionExec, | ||
| CometFilterExec, CometProjectionExec, MembershipKeyReductionExec, SortInputCoalesceExec, | ||
| }; | ||
| use datafusion_comet_proto::{ | ||
| spark_expression::Expr, | ||
|
|
@@ -220,7 +220,8 @@ fn reduce_before_sort( | |
| let Some(sort) = params.right.native_plan.downcast_ref::<SortExec>() else { | ||
| return Ok(None); | ||
| }; | ||
| let input = sort.input(); | ||
| // The reducer packs its own output, so it replaces any input coalescer. | ||
| let input = SortInputCoalesceExec::unwrap_input(sort.input()); | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could we keep the coalescer between the reducer and the sort when both options are enabled? The reducer only packs the output of a successful reduction. For the 62-row shuffle blocks described in the PR, every block is below its 1,024-row minimum, so the reducer forwards each one separately. Enabling right-key reduction on an eligible left-semi join would therefore restore the original merge fan-in on the right sort. A test with both flags and a long stream of short blocks could check that bypassed rows still reach the sort in combined batches. Reducer decision and forwarding |
||
| let ([(_, key)], [options]) = (params.join_on.as_slice(), sort_options) else { | ||
| return Ok(None); | ||
| }; | ||
|
|
@@ -268,7 +269,11 @@ fn reduce_before_sort( | |
| vec![Arc::clone(&reducer)], | ||
| ReplaceChildrenOptions::new(ChildrenPropertiesMode::Recompute), | ||
| )?; | ||
| Arc::make_mut(&mut params.right).native_plan = sort; | ||
| let right = Arc::make_mut(&mut params.right); | ||
| right.native_plan = sort; | ||
| right | ||
| .additional_native_plans | ||
| .retain(|plan| !plan.is::<SortInputCoalesceExec>()); | ||
| // The Spark sort collects from its replacement. The join owns the reducer's named metrics. | ||
| Ok(Some(reducer)) | ||
| } | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since #7630 landed on
main,CometFairMemoryPoolsplits the share only among consumers that can spill and are active. A sort that has not started takes no share from another sort (is_sortandunstarted_sortsintry_grow). ThisExternalSorter[1]registers but never reserves. So after a merge withmain, I expect the sort under test to get the whole 3 MiB share, and the test stops covering the halved share it describes. It should still pass, because the plain sort spills either way, so CI will not flag it.hold(&other_sort, 2)frommemory_pools/testing.rsmakes it active, as the tests added in #7630 do. The finalpool.reserved()check then has to allow for those 2 bytes. Untested.