Repository navigation
Conversation
This was referenced Nov 15, 2026
Arrow's offsets buffers for Utf8, Binary, list and map columns hold running sums, which zstd compresses poorly. With the zstd codec, shuffle writers now rewrite each batch's offsets as value lengths split into byte planes and compress the body as one zstd frame. Readers turn those messages back into standard IPC before arrow-ipc decodes them, and pass every other message through unchanged. The rewritten record-batch messages use a BodyCompression codec Arrow does not define, so readers that don't know the format fail instead of misreading the data. `ballista.shuffle.compression.offsets_as_lengths` (default true) turns it off. Closes apache#2804 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
andygrove
force-pushed
the
perf/offsets-as-lengths
branch
from
November 22, 2026 17:29
b156ad2 to
fbdf495
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #2804.
Rationale for this change
Shuffle data is Arrow IPC with arrow-ipc's per-buffer compression. The offsets buffers of
Utf8,Binary, list and map columns hold running sums. Those never repeat, so zstd compresses them poorly even when every value has the same length, and they often cost about as much as the string bytes. Stored as value lengths split into byte planes, they compress to almost nothing. DataFusion Comet made the same change in apache/datafusion-comet#6943.shuffle_bench, one task, 16 hash partitions, 8,192-row batches, zstd at the shuffle's level 1 in both columns, TPC-H SF100 tables with all columns, two interleaved runs each:Writing and reading back are no slower, and slightly faster with smaller output: lineitem write 11.23 s / 11.25 s off vs 10.81 s / 10.90 s on, read back 2.70 s / 2.78 s off vs 2.57 s / 2.60 s on; part write 7.50 s / 7.47 s off vs 7.09 s / 7.05 s on. At level 3 the savings were 1-2 points larger (customer 17.8%, part 33.0%, orders 23.6%, lineitem 23.9%) for about 1 s more write time on lineitem. These are single-machine numbers. Cluster runs are still to do.
What changes are included in this PR?
ballista_core::shuffle_ipc(new):offsets.rsports Comet's buffer-role mapping and the in-place rewrite between offsets and byte-plane lengths.ShuffleStreamWriterencodes each batch uncompressed withIpcDataGenerator, rewrites the offsets, compresses the body as one zstd frame (atSHUFFLE_ZSTD_COMPRESSION_LEVEL, level 1, like the plain path) and rebuilds the message.ShuffleEncodingdecides once per schema whether to do that: only with zstd, only for schemas with an offsets buffer whose types all have a mapped layout, and only when arrow-ipc lays out an empty batch of the schema as the mapping expects. Everything else gets arrow-ipc'sStreamWriterand identical bytes.ShuffleIpcReader(aReadadapter forStreamReader) andShuffleBytesTranslator(push-based, for block transport) turn rewritten messages back into standard uncompressed IPC messages and pass all other messages through byte for byte.Format: the stream keeps Arrow IPC framing. A rewritten record-batch message has a
BodyCompressioncodec Arrow does not define (0x40), buffer offsets that describe the uncompressed body, and a body holding the uncompressed and compressed lengths followed by the zstd frame. Readers that do not know it fail with arrow-ipc's "compression type not supported" error instead of misreading the data. Each stream stays self-contained, so the sort shuffle's byte-copied spill streams and its index are unchanged.Writers: the passthrough
ShuffleWriterExec(now throughwrite_shuffle_stream_to_disk), sort shuffle spills and the sort shuffle's in-memory partition encoding. The encoded partitions that the memory pool refuses and that go to the spill file instead (fix(core): reserve the sort-shuffle writer's encoded output #2803) use the same encoding, so both paths still write identical bytes.Readers: local passthrough reads,
MultiStreamPartitionStream, the Flightdo_getserve path (which still re-encodes with LZ4, so clients are unaffected) andBlockDataStream.Not covered: range shuffle (IPC file format with per-message slicing), pipelined in-memory shuffle, and LZ4 (lz4_flex decodes long runs slowly, perf: faster overlapping copy in the decoder PSeitz/lz4_flex#227).
ballista.shuffle.compression.offsets_as_lengths(defaulttrue, only has an effect with zstd) as an off switch.No separate protocol bump: block transport ships raw shuffle bytes to the consuming executor, so older executors can't read them, but perf: deduplicate executor metadata in partition locations #2813 already raised
BALLISTA_PROTOCOL_VERSIONto 4 in this release, which keeps 55.x executors out.BlockDataStreamnow fails when its input ends inside a message, instead of ending cleanly. The translator holds a rewritten batch back until its whole body has arrived, so without this a truncated transfer would drop that batch silently.Memory: the rewritten path holds a batch's uncompressed body and its compressed frame at the same time, where arrow-ipc compresses buffer by buffer. That is about one extra batch of transient memory per writer and per reader, outside the memory pool.
shuffle_benchgains--codecand--offsets-as-lengths, and reports shuffle bytes and read-back time.Tests: Comet's role and round-trip unit tests, plus round trips of string, binary, list, large list, fixed-size list, map and struct columns with nulls, empty values, sliced inputs and empty batches under every codec and setting. They cover both readers, with block-transport bytes fed in blocks of 1 to N bytes, and check that:
A
BlockDataStreamtest checks that a stream cut inside a rewritten batch is an error.The existing
sort_shuffleend-to-end matrix (local, Flight, block transport) passes with the setting on by default.Are there any user-facing changes?
Yes:
ballista.shuffle.compression.offsets_as_lengthssetting.BALLISTA_PROTOCOL_VERSION4 already in this release covers that, so the scheduler and all executors must be upgraded together.ShuffleIpcReaderorShuffleBytesTranslatorin front of them.ballista_core::shuffle_ipcmodule,utils::write_shuffle_stream_to_disk,SpillManager::with_encodingandBallistaConfig::shuffle_compression.write_stream_to_diskandSpillManager::newkeep their signatures and still write plain arrow-ipc streams.The upgrade guide has an entry for this.