Skip to content

perf: combine small input batches before a native sort that merges them - #7696

Open
parthchandra wants to merge 1 commit into
apache:mainfrom
parthchandra:sort-coalesce-shuffle-read
Open

parthchandra wants to merge 1 commit into
apache:mainfrom
parthchandra:sort-coalesce-shuffle-read

Conversation

@parthchandra

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #5686.

Rationale for this change

A shuffle written by many map tasks into many partitions gives each reduce task one tiny Arrow batch per shuffle block. On a production job, about 330,000 map tasks write to 16,000 partitions, so each reduce task reads about 330,000 blocks of about 62 rows. When the native sort over that input spills, it is extremely slow. In one task, SortExec took 16,778 s of elapsed_compute for 19.7 M rows and 5 spills, about 97% of the task. The shuffle read took 33 s.

The cause is DataFusion 55.2's ExternalSorter:

  • When it spills, sort_and_spill_in_mem_batches calls in_mem_sort_stream(false, false), which does not combine buffered batches into runs. Every buffered batch becomes its own merge stream, so a spill of about 4 M rows merges about 66,000 streams.
  • For every output batch, BatchBuilder calls arrow interleave over every live input batch, for each column and each struct child. The cost per output batch grows with the number of streams.
  • Without spilling, DataFusion combines runs only for single-key sorts. Multi-key sorts merge one stream per batch either way.

#5686 found the same problem and measured a large native gain. It was set aside after an end-to-end run regressed by 11%. In that run each partition held about 7,800 rows, under the 1 MiB sort_in_place_threshold_bytes, so the sort never merged: it concatenated in place, and the extra CoalesceBatchesExec was pure overhead. The issue also pointed out that the coalescer's buffers were not charged to a memory pool. This PR follows the analysis in #5686 and addresses both points.

What changes are included in this PR?

  • SortInputCoalesceExec (native/operators/src/sort_input_coalesce.rs), a new operator in front of a native SortExec:
    • Gate: batches pass through unchanged until a task's sort input reaches the sort's in-place threshold, counted the way the sort counts it. Small sorts, the case that regressed in perf: coalesce small shuffle-read batches before native sort #5686, are unaffected.
    • Combining: after the gate, batches with fewer rows than half of spark.comet.batchSize and fewer than 512 are combined in arrival order. A combined batch holds at most spark.comet.batchSize rows and half the in-place threshold in bytes, 512 KiB by default, kept between 64 KiB and 4 MiB. The sort then reserves no more for a combined batch than its own in-place threshold. Input batches are never split.
    • Memory: it reserves exactly what it holds, the batches plus the combined copy, in a consumer that cannot spill. It releases the reservation before the combined batch reaches the sort, so it holds nothing while the sort reserves. If the pool refuses, it hands the group and the batch to the sort and passes the next 64 small batches through before asking again. Errors other than ResourcesExhausted are returned.
    • Never combined: batches with a different schema start a new group. Batches with small dictionary keys or small run-end types, at any nesting depth, pass through, since concatenating them can overflow the key type. A small slice of a large batch counts as large and passes through.
    • Metrics: coalesced_input_batches and coalesced_output_batches are reported under the sort. The operator's time is part of the sort's time.
  • Planner: the Sort arm adds the coalescer only when Sort.coalesce_input (a new proto field) is set, the sort has no fetch (TopK never merges), and the input does not already satisfy the ordering (the sort passes such input through). The semi-join key reduction in join.rs looks through the coalescer and replaces it.
  • Config: spark.comet.exec.sort.coalesceInput.enabled, on by default. CometSortExec sets the proto field from it and declares the two metrics.
  • Docs:
    • The migration guide gets a 1.3.0 note.
    • metrics.md gets a CometSort section.
    • memory_management.md describes the reservation and the trade-off below.
  • Benchmark: native/operators/benches/sort_small_batches.rs.

Trade-off to review. A combined batch is larger than the blocks it replaces. A sort whose memory share is very close to the least it can run with can now fail with "Not enough memory to continue external sort" where the tiny blocks fit. The same is already true of any input that delivers full-size batches. Halving the cap and holding nothing while the sort reserves made this much narrower: in tight-pool sweeps, the coalesced sort spilled slightly less than the plain one and never failed where it fit. It is documented, with the config as the way out. The complete fix belongs in DataFusion, as a follow-up below.

Follow-ups, not in this PR:

  • In DataFusion, combine buffered batches into runs on the spill path, within the memory budget. Also have ExternalSorter split a batch it cannot reserve after spilling, instead of failing. Either would make this operator unnecessary or safer.
  • Hold the input before the gate without copying it, so the start of each task's input is combined too.

How are these changes tested?

  • Rust unit tests in sort_input_coalesce.rs (28):
    • Gate and grouping: pass-through below the gate without a copy; the gate's byte counting; the row target, byte cap and 512-row limit; large batches flushing the group first; empty, zero-row and zero-column batches.
    • Memory: the reservation covers exactly the held batches and is released; nothing is held while the sort receives a batch; a refused reservation flushes the held group first; backoff and resumption; combining when the pool fits the group but not a full one.
    • Errors and early drop: an input error reaches the sort and releases memory; dropping the stream early releases the reservation.
    • Batches never combined: schema changes; small dictionary keys at the top level, in a struct and in a list; slices of large batches.
    • Metrics: a rebuilt node reports through the original's metrics.
    • Types: a nested map, list and struct mix, decimal(38,10) at its limits, years 1 and 9999, a time zone, CJK and other non-ASCII strings, and dictionaries.
    • Spilling sorts: single-key and multi-key spilling SortExecs give identical output with and without the coalescer.
  • CometFairMemoryPool test (fair_pool.rs): a sort whose fair share is halved by a second spillable consumer spills and finishes with combined input wherever it does without. It failed with the earlier 1 MiB cap.
  • Planner tests (4): only a full sort over unordered input is wrapped, with the session's batch size and threshold; a sort with only a skip is wrapped; the semi-join key reduction replaces the coalescer.
  • sql-tests/operators/sort_coalesce_input.sql: compared with Spark. Each sort task reads blocks of about 12 rows.
    • The data covers nested structs, maps and arrays of structs, decimal bounds, dates, timestamps and timestamp_ntz from year 1 to 9999, and non-ASCII strings.
    • Sort keys include a single string key, multiple keys with both null orderings, and a struct field.
    • The config matrix covers the in-place threshold (0 and the default), a normal and a tiny memory pool so the sort spills, and two session time zones.
  • CometTaskMetricsSuite: a sort over 80 map tasks' small blocks spills and matches Spark. With the config on it combines batches; with it off it does not. The test checks metrics, which a SQL file cannot.
  • Micro-benchmark sort_small_batches: 200,000 wide rows (fixed-length string key, 8 longs, a struct, no nulls), one or two sort keys, in memory and with a memory limit that forces spills, with keys scattered so every batch spans the whole key range (poorly predicted) and with keys already in order (the most predictable merge). Times in ms, raw -> combined, on a laptop with about +/-10% run-to-run noise. Scattered keys:
Rows per input batch In memory, 1 key In memory, 2 keys Spilling, 1 key Spilling, 2 keys
16 86.3 -> 51.9 330.2 -> 51.2 256.2 -> 79.3 258.2 -> 70.7
64 38.8 -> 31.9 97.5 -> 32.0 102.0 -> 58.6 97.7 -> 49.8
256 32.1 -> 29.7 45.1 -> 29.0 64.2 -> 57.4 62.6 -> 50.3
384 31.2 -> 28.3 38.7 -> 28.3 58.0 -> 57.0 55.3 -> 51.2
1024 (passed through) 27.0 -> 27.2 31.0 -> 31.1 40.1 -> 42.7 48.3 -> 45.7

Keys already in order:

Rows per input batch In memory, 1 key In memory, 2 keys Spilling, 1 key Spilling, 2 keys
16 66.2 -> 38.3 310.0 -> 47.1 221.0 -> 68.3 259.0 -> 73.2
64 29.2 -> 22.1 90.7 -> 26.9 76.6 -> 47.4 99.1 -> 56.2
256 19.7 -> 18.1 35.2 -> 23.4 44.7 -> 42.5 50.1 -> 48.2
1024 (passed through) 16.9 -> 17.1 21.2 -> 20.9 34.5 -> 34.3 40.0 -> 47.2

When 512-row batches were combined, a spilling single-key sort got slower (51.4 -> 54.0 ms), so the limit is fewer than 512 rows. Multi-key sorts gain the most, even in memory, where DataFusion does not combine runs itself.

Cost: batches the coalescer passes through are unchanged, but the extra operator is not free. With keys in order, spilling and two keys, 1024-row input is about 7 ms (~18%) slower with the coalescer in place, with the same spill count. An identity ProjectionExec in the same place costs the same (38.6 -> 44.3 ms), so this is the cost of one more operator between an in-memory source and the sort, not of the coalescer's work. In a Comet plan the sort's input is already a shuffle reader or scan. Sorts whose input is declared in order are not wrapped. In the TPC-H runs below, total time did not get worse; q7 is the one query where it might show.

  • End-to-end: TPC-H SF100 on one machine (Spark 4.0 standalone, 2 executors x 8 cores), with spark.comet.exec.sort.coalesceInput.enabled off and on, spark.sql.shuffle.partitions=2000 and AQE partition coalescing off, at 16g and at 2g off-heap. Three iterations each; times are the average.
Off-heap Off (s) On (s) Change Failed tasks, on
16g 1148.3 1144.9 -0.3% 0
2g 1257.7 1234.8 -1.8% 0
  • The coalescer combined batches in q3, q4, q9, q10 and q21, including about 8.4 M small batches in q9 and 5.2 M in q10. Those queries changed by 1% or less: their sorts did not spill and are not where the time goes. TPC-H does not reach the spill-heavy sort over tiny blocks that perf: coalesce small shuffle-read batches before native sort #5686 describes, so this run shows that the change costs nothing on a standard workload, not the speed-up.
  • No query failed with the coalescer on, and spill counts did not change. The only failures were in the run with it off: an executor was lost during one q5 iteration (256.8 s against about 87 s for the others), which is left out of the 16g "off" total above (1204.9 s with it).
  • q11 (about 3 s or 25 s per iteration whatever the setting) and q12 (82 to 112 s) vary too much between iterations to compare. q7 was about 10% slower with the coalescer on at 16g, though nothing was combined there. That could be the cost of the extra operator described above, or noise; three iterations do not separate the two.

A shuffle from many map tasks gives each reduce task one tiny batch per
block. DataFusion's sort merges one stream per buffered batch when it
spills, so such sorts become very slow.

Add SortInputCoalesceExec in front of a full native sort over unordered
input. It passes batches through until the input reaches the sort's
in-place threshold, then combines small batches into batches of at most
spark.comet.batchSize rows and half that threshold in bytes. It reserves
exactly what it holds and holds nothing while the sort takes a batch.

Controlled by spark.comet.exec.sort.coalesceInput.enabled (default
true). Closes apache#5686.
@parthchandra
parthchandra requested review from andygrove, comphead, mbutrovich and sunchao and removed request for andygrove November 19, 2026 21:39
@github-actions github-actions Bot added enhancement New feature or request performance area:memory Memory pools, reservations, OOM handling labels Nov 19, 2026
@parthchandra parthchandra added run-spark-4.0-tests run-benchmark-check Run the benchmark compile and lint check on this pull request instead of waiting for the merge queue run-all-spark-profiles Run the Comet test suites against every Spark profile on this pull request, ahead of the merge queue labels Nov 19, 2026

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked the operator against DataFusion 55.2's ExternalSorter. It reserves before it holds a batch and frees before the sort reserves, and the planner wraps only full sorts over unordered input. One fair_pool.rs test no longer tests what it says once merged with #7630, and a few tests repeat others. For the DataFusion follow-up, the two-key in-memory numbers are worth including: in_mem_sort_stream in 55.2 and 56.0 keeps multi-column sorts at one run per batch, on the grounds that merging many small runs is faster, and sort_small_batches shows the opposite for tiny batches.

Comment on lines +413 to +414
// Another consumer that can spill, such as a second sort in the task, halves the share.
let _other_sort = spillable("ExternalSorter[1]", &pool);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since #7630 landed on main, CometFairMemoryPool splits the share only among consumers that can spill and are active. A sort that has not started takes no share from another sort (is_sort and unstarted_sorts in try_grow). This ExternalSorter[1] registers but never reserves. So after a merge with main, I expect the sort under test to get the whole 3 MiB share, and the test stops covering the halved share it describes. It should still pass, because the plain sort spills either way, so CI will not flag it. hold(&other_sort, 2) from memory_pools/testing.rs makes it active, as the tests added in #7630 do. The final pool.reserved() check then has to allow for those 2 bytes. Untested.

Comment on lines +680 to +722
#[test]
fn a_refused_reservation_flushes_the_held_group_first() {
let input = ten_row_batches(4);
let one = get_record_batch_memory_size(&input[0]);
// Room for two batches and their copy. The third is refused, so the first two go out
// combined and the third passes through. The fourth starts a new group.
let output = run_in_pool(input.clone(), 1000, 2 * 2 * one);
assert_eq!(row_counts(&output), [20, 10, 10]);
same_rows(&output, &input);
}

#[tokio::test]
async fn holds_no_memory_while_the_sort_takes_a_batch() {
let input = ten_row_batches(10);
let schema = input[0].schema();
let pool: Arc<dyn MemoryPool> = Arc::new(GreedyMemoryPool::new(1 << 30));
let runtime = RuntimeEnvBuilder::new()
.with_memory_pool(Arc::clone(&pool))
.build_arc()
.unwrap();
let ctx = SessionContext::new_with_config_rt(SessionConfig::new(), runtime);
let source = MemorySourceConfig::try_new_exec(&[input], schema, None).unwrap();
// A 25-row target makes every third batch overflow the group.
let exec = SortInputCoalesceExec::new(source, 25, 0);
let mut stream = exec.execute(0, ctx.task_ctx()).unwrap();
let mut rows = vec![];
while let Some(batch) = stream.try_next().await.unwrap() {
assert_eq!(pool.reserved(), 0, "held memory while handing on a batch");
rows.push(batch.num_rows());
}
assert_eq!(rows, [20, 20, 20, 20, 20]);
}

#[test]
fn after_a_refusal_the_next_batches_pass_through() {
let input = ten_row_batches(6);
let one = get_record_batch_memory_size(&input[0]);
// Room for two batches and their copy, so the third is refused. The ones after it pass
// through without asking the pool again.
let output = run_in_pool(input.clone(), 1000, 2 * 2 * one);
assert_eq!(row_counts(&output), [20, 10, 10, 10, 10]);
same_rows(&output, &input);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two tests use the same pool size and target as combining_resumes_after_the_backoff below. Their expected outputs, [20, 10, 10] and [20, 10, 10, 10, 10], are prefixes of its output, so that test alone covers all three. Also, the comment in the first one is off: the fourth batch passes through under the backoff and does not start a new group. The three dictionary tests at lines 767 to 816 differ only in how the column wraps the Int8 dictionary. One test that loops over the three columns would cover them.

Comment on lines +979 to +1044
/// A sort in a fair pool so tight that it spills about every 50 KiB. It failed with combined
/// input when the coalescer still held memory while the sort reserved a combined batch.
async fn sort_in_tight_fair_pool(
input: Vec<RecordBatch>,
coalesce: bool,
) -> Result<MetricsSet> {
use datafusion::execution::memory_pool::FairSpillPool;
use datafusion::physical_expr::expressions::col;
use datafusion::physical_expr::{LexOrdering, PhysicalSortExpr};
use datafusion::physical_plan::sorts::sort::SortExec;

let schema = input[0].schema();
let runtime = RuntimeEnvBuilder::new()
.with_memory_pool(Arc::new(FairSpillPool::new(2560 * 1024)))
.build_arc()
.unwrap();
let config = SessionConfig::new().with_sort_spill_reservation_bytes(64 * 1024);
let gate = config.options().execution.sort_in_place_threshold_bytes;
let ctx = SessionContext::new_with_config_rt(config, runtime);
let source = MemorySourceConfig::try_new_exec(&[input], Arc::clone(&schema), None).unwrap();
let coalescer = Arc::new(SortInputCoalesceExec::new(
Arc::clone(&source) as _,
8192,
gate,
));
let sort_input: Arc<dyn ExecutionPlan> = if coalesce {
Arc::clone(&coalescer) as _
} else {
source
};
let ordering = LexOrdering::new(vec![PhysicalSortExpr::new(
col("id", &schema).unwrap(),
Default::default(),
)])
.unwrap();
let sort = Arc::new(SortExec::new(ordering, sort_input));
collect(sort.execute(0, ctx.task_ctx())?).await?;
Ok(coalescer.metrics().unwrap())
}

#[tokio::test]
async fn combined_input_does_not_fail_a_sort_in_a_tight_fair_pool() {
let input: Vec<_> = (0..4000)
.map(|b| {
let ids: Vec<i64> = (0..50)
.map(|i| ((b * 50 + i) * 7919 % 200_000) as i64)
.collect();
let text = ids.iter().map(|i| format!("value-{i:010}-padding-padding"));
RecordBatch::try_from_iter([
("id", Arc::new(Int64Array::from(ids.clone())) as ArrayRef),
(
"s",
Arc::new(StringArray::from_iter_values(text)) as ArrayRef,
),
])
.unwrap()
})
.collect();
sort_in_tight_fair_pool(input.clone(), false)
.await
.expect("the plain sort fits");
let metrics = sort_in_tight_fair_pool(input, true)
.await
.expect("the sort over combined batches fits too");
assert!(metric(&metrics, "coalesced_input_batches") > 0);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This runs the same scenario as a_sort_fits_its_share_with_combined_input_where_it_fits_without in fair_pool.rs: a sort over small batches in a tight fair pool, with and without the coalescer. The difference is that it uses DataFusion's FairSpillPool, which Comet does not use outside tests. The failure it describes, the coalescer holding memory while the sort reserves, is the invariant that holds_no_memory_while_the_sort_takes_a_batch checks directly. I think this test and sort_in_tight_fair_pool can be removed.

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd keep spark.comet.exec.sort.coalesceInput.enabled opt-in for now. I traced two independent memory problems through DataFusion 55.2 and Arrow 59.3.2. A group can fit the coalescer's budget and still fail the sort's retry after a spill. Shared input columns can also expand during concatenation and exceed the group's byte cap. I've left concrete inputs for both inline. There's an open upstream PR that forms runs inside the sorter and retains the original batches if the replacement fails admission. That seems like a good place to exercise the same inputs. DataFusion #26931

Before enabling this by default, could we get paired Spark stage timings with the flag off and on for the spilling small-block workload and for the 128-writer/128-reducer case from #5686? The TPC-H sorts that combined blocks stayed in memory. On the small-reducer path, each batch now goes through a second size calculation until the gate opens. Those two measurements would show the gain on the target stage and the cost below the gate. Prior Spark measurements

context: Arc<TaskContext>,
) -> Result<SendableRecordBatchStream> {
let mut input = self.input.execute(partition, Arc::clone(&context))?;
let reservation = MemoryConsumer::new(format!("SortInputCoalesceExec[{partition}]"))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pool can grant this unspillable consumer a group that exceeds ExternalSorter’s fair share. That can fail a task even when 24 raw batches fit at once. Fair-pool checks · Sort retry

A regression case for the new fair-pool test would be an 11.5 MiB pool, a second active spillable consumer, and the default 10 MiB merge reservation, 1 MiB threshold and 8,192-row target. Feed it 64 batches of 256 rows with eight separately allocated, non-null Int64 columns. Each batch holds 16 KiB and costs the sort 32 KiB. Tracing the reservations through 55.2 gives the following sequence.

  • With coalescing off, the 768 KiB sort share admits runs of 24, 24 and 16 batches. Their spill files fit in the merge workspace.
  • With coalescing on, the gate opens after batch 32. The sort holds 256 KiB at that point. The pool grants the coalescer 1 MiB for batches 33–64 and their copy. Once emitted, that single 512 KiB batch costs the sort 1 MiB. The sort spills its 256 KiB and retries with an empty buffer, where the limit is still 768 KiB.

The backoff only sees refusals on the coalescer's reservation. Could we add this case and keep the feature opt-in until an oversized group can fall back to the original batches under the sort's own reservation?

self.backoff -= 1;
None
} else {
Some(get_record_batch_memory_size(batch))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we budget retained input and materialized output separately here? get_record_batch_memory_size counts each buffer address once. A projection such as named_struct('a', s, 'b', s, ...) can reuse the same string array for several fields. Arrow concatenates each field into a separate array. Struct evaluation · Input accounting · Arrow concatenation

For example, after the gate, take two 256-row batches, each with a 128 KiB string buffer shared by eight fields. Together the inputs fit the 512 KiB cap. We reserve about 512 KiB, then allocate 2 MiB for the strings alone while retaining the inputs. The sort then needs about 4 MiB to admit the result. Counting the output per field before concatenating would let us cap the copy and reserve for both allocations. A test with repeated Arc::clones of one StringArray would catch this.

if needed <= size {
return Ok(true);
}
match self.reservation.try_grow(needed - size) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This request can spill a JVM operator before we know whether there is a second batch to combine. With fair_unified and the sort at its share, the original sort request hits the local limit and triggers a native spill. The unspillable coalescer goes straight to Spark, which can spill a resident UnsafeExternalSorter to grant the request. That disk write has already happened even if the next batch forces a one-batch flush. We measured this behavior with a temporary reservation in #7190. Prior spill measurement

Could we apply the same resident-memory guard used for semi-join reduction, or charge this request against the sort's remaining share before calling Spark? The existing admission test could exercise this with small blocks and a sibling that counts spill calls. Planner guard · Spark test

};
let input = sort.input();
// The reducer packs its own output, so it replaces any input coalescer.
let input = SortInputCoalesceExec::unwrap_input(sort.input());

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we keep the coalescer between the reducer and the sort when both options are enabled? The reducer only packs the output of a successful reduction. For the 62-row shuffle blocks described in the PR, every block is below its 1,024-row minimum, so the reducer forwards each one separately. Enabling right-key reduction on an eligible left-semi join would therefore restore the original merge fan-in on the right sort. A test with both flags and a long stream of short blocks could check that bypassed rows still reach the sort in combined batches. Reducer decision and forwarding

columns.push((
"g".into(),
Arc::new(Int64Array::from_iter_values(
ids.iter().map(|&i| (i % 64) as i64),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The two_keys/in_order case sorts by (g, key), so the leading key wraps from 63 to 0 at i = 64. At 64 rows per batch every run spans all 64 groups, and at 1,024 rows even a single batch wraps repeatedly. The reported 18% slowdown therefore measures a cyclic leading key. Could we generate contiguous groups with i * 64 / ROWS, check lexicographic order across batch boundaries outside the timed loop, and refresh the two-key numbers?

grouper.prepare(&batch)?
};
if let Some(batch) = flushed {
baseline.record_output(batch.num_rows());

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we record the emitted batch with (&batch).record_output(&baseline) on all three emission paths, as ExplodeExec does? Passing the row count updates only output_rows. The native plan then reports zero output_batches and output_bytes even when coalesced_output_batches is positive. The total batch count is particularly useful here because it includes the blocks passed through before the gate and tells us how many batches actually reached the sort. DataFusion counters · Comet example

let exec = SortInputCoalesceExec::new(source, 40, 0);
let mut stream = exec.execute(0, ctx.task_ctx()).unwrap();
// The first group is out and the second is being held.
assert_eq!(stream.try_next().await.unwrap().unwrap().num_rows(), 40);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At this point flush() has freed the first group's reservation and the stream is suspended in emit().await, before it has pulled the next group. The pool is already at zero before drop(stream). Could we have the input yield two short batches and then return Poll::Pending? Polling the coalescer once and asserting pool.reserved() > 0 before dropping it would exercise cancellation with a live reservation.

-- The second run gives the sort a tiny memory pool and merge reservation so that it spills.
-- Config: spark.comet.datafusion.execution.sort_spill_reservation_bytes=65536
-- ConfigMatrix: spark.comet.exec.memoryPool.fraction=1.0,0.002
-- ConfigMatrix: spark.sql.session.timeZone=UTC,America/Los_Angeles

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we keep this file focused on Spark parity for nested values with one configuration and a smaller input? The three matrix dimensions rerun the 200-file setup and six queries eight times. The test durations on this commit sum to 5m58s on Spark 4.1 and 26m56s across the five profiles. Spark 4.1 timings · Other profiles

The new CometTaskMetricsSuite test already verifies that the sort combines batches after the default gate, spills, and honors the toggle. Setting the threshold to zero here would let a handful of small files exercise concatenation from the first block. One time zone and one pool size should cover the nested copy. Metrics test

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:memory Memory pools, reservations, OOM handling enhancement New feature or request performance run-all-spark-profiles Run the Comet test suites against every Spark profile on this pull request, ahead of the merge queue run-benchmark-check Run the benchmark compile and lint check on this pull request instead of waiting for the merge queue run-spark-4.0-tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf: coalesce small shuffle-read batches before native sort

3 participants