Skip to content

perf: store shuffle string offsets as lengths before zstd compression (experimental) - #2809

Draft
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:perf/offsets-as-lengths
Draft

andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:perf/offsets-as-lengths

Conversation

@andygrove

@andygrove andygrove commented Nov 8, 2026 •

Copy link
Copy Markdown
Member

Experimental / WIP. Opened as a draft to get early feedback on the format and the approach. Not ready to merge.

Which issue does this PR close?

Closes #2804.

Rationale for this change

Shuffle data is Arrow IPC with arrow-ipc's per-buffer compression. The offsets buffers of Utf8, Binary, list and map columns hold running sums. Those never repeat, so zstd compresses them poorly even when every value has the same length, and they often cost about as much as the string bytes. Stored as value lengths split into byte planes, they compress to almost nothing. DataFusion Comet made the same change in apache/datafusion-comet#6943.

shuffle_bench, one task, 16 hash partitions, 8,192-row batches, zstd at the shuffle's level 1 in both columns, TPC-H SF100 tables with all columns, two interleaved runs each:

Table (rows) Offsets as is Offsets as lengths Smaller by
customer (15M) 980.0 MB 823.5 MB 16.0%
part (20M) 814.6 MB 559.1 MB 31.4%
orders (first 30M) 1,127.4 MB 873.9 MB 22.5%
lineitem (first 30M) 1,262.8 MB 982.2 MB 22.2%

Writing and reading back are no slower, and slightly faster with smaller output: lineitem write 11.23 s / 11.25 s off vs 10.81 s / 10.90 s on, read back 2.70 s / 2.78 s off vs 2.57 s / 2.60 s on; part write 7.50 s / 7.47 s off vs 7.09 s / 7.05 s on. At level 3 the savings were 1-2 points larger (customer 17.8%, part 33.0%, orders 23.6%, lineitem 23.9%) for about 1 s more write time on lineitem. These are single-machine numbers. Cluster runs are still to do.

What changes are included in this PR?

  • ballista_core::shuffle_ipc (new):

    • offsets.rs ports Comet's buffer-role mapping and the in-place rewrite between offsets and byte-plane lengths.
    • ShuffleStreamWriter encodes each batch uncompressed with IpcDataGenerator, rewrites the offsets, compresses the body as one zstd frame (at SHUFFLE_ZSTD_COMPRESSION_LEVEL, level 1, like the plain path) and rebuilds the message. ShuffleEncoding decides once per schema whether to do that: only with zstd, only for schemas with an offsets buffer whose types all have a mapped layout, and only when arrow-ipc lays out an empty batch of the schema as the mapping expects. Everything else gets arrow-ipc's StreamWriter and identical bytes.
    • ShuffleIpcReader (a Read adapter for StreamReader) and ShuffleBytesTranslator (push-based, for block transport) turn rewritten messages back into standard uncompressed IPC messages and pass all other messages through byte for byte.
  • Format: the stream keeps Arrow IPC framing. A rewritten record-batch message has a BodyCompression codec Arrow does not define (0x40), buffer offsets that describe the uncompressed body, and a body holding the uncompressed and compressed lengths followed by the zstd frame. Readers that do not know it fail with arrow-ipc's "compression type not supported" error instead of misreading the data. Each stream stays self-contained, so the sort shuffle's byte-copied spill streams and its index are unchanged.

  • Writers: the passthrough ShuffleWriterExec (now through write_shuffle_stream_to_disk), sort shuffle spills and the sort shuffle's in-memory partition encoding. The encoded partitions that the memory pool refuses and that go to the spill file instead (fix(core): reserve the sort-shuffle writer's encoded output #2803) use the same encoding, so both paths still write identical bytes.

  • Readers: local passthrough reads, MultiStreamPartitionStream, the Flight do_get serve path (which still re-encodes with LZ4, so clients are unaffected) and BlockDataStream.

  • Not covered: range shuffle (IPC file format with per-message slicing), pipelined in-memory shuffle, and LZ4 (lz4_flex decodes long runs slowly, perf: faster overlapping copy in the decoder PSeitz/lz4_flex#227).

  • ballista.shuffle.compression.offsets_as_lengths (default true, only has an effect with zstd) as an off switch.

  • No separate protocol bump: block transport ships raw shuffle bytes to the consuming executor, so older executors can't read them, but perf: deduplicate executor metadata in partition locations #2813 already raised BALLISTA_PROTOCOL_VERSION to 4 in this release, which keeps 55.x executors out.

  • BlockDataStream now fails when its input ends inside a message, instead of ending cleanly. The translator holds a rewritten batch back until its whole body has arrived, so without this a truncated transfer would drop that batch silently.

  • Memory: the rewritten path holds a batch's uncompressed body and its compressed frame at the same time, where arrow-ipc compresses buffer by buffer. That is about one extra batch of transient memory per writer and per reader, outside the memory pool.

  • shuffle_bench gains --codec and --offsets-as-lengths, and reports shuffle bytes and read-back time.

  • Tests: Comet's role and round-trip unit tests, plus round trips of string, binary, list, large list, fixed-size list, map and struct columns with nulls, empty values, sliced inputs and empty batches under every codec and setting. They cover both readers, with block-transport bytes fed in blocks of 1 to N bytes, and check that:

    • plain streams match arrow-ipc's byte for byte,
    • unmapped schemas (views, dictionaries) and schemas without offsets fall back,
    • concatenated streams are crossed correctly,
    • an unaware arrow reader rejects the format,
    • truncated or unmapped rewritten batches are errors.

    A BlockDataStream test checks that a stream cut inside a rewritten batch is an error.

    The existing sort_shuffle end-to-end matrix (local, Flight, block transport) passes with the setting on by default.

Are there any user-facing changes?

Yes:

  • Shuffle files written with zstd are smaller for schemas with string, binary, list or map columns.
  • The new ballista.shuffle.compression.offsets_as_lengths setting.
  • Shuffle data with offsets as lengths can't be read by executors from earlier releases. The bump to BALLISTA_PROTOCOL_VERSION 4 already in this release covers that, so the scheduler and all executors must be upgraded together.
  • Code of your own that reads shuffle files with arrow-ipc's readers needs ShuffleIpcReader or ShuffleBytesTranslator in front of them.
  • New public APIs, no signature changes: the ballista_core::shuffle_ipc module, utils::write_shuffle_stream_to_disk, SpillManager::with_encoding and BallistaConfig::shuffle_compression. write_stream_to_disk and SpillManager::new keep their signatures and still write plain arrow-ipc streams.

The upgrade guide has an entry for this.

Arrow's offsets buffers for Utf8, Binary, list and map columns hold
running sums, which zstd compresses poorly. With the zstd codec, shuffle
writers now rewrite each batch's offsets as value lengths split into
byte planes and compress the body as one zstd frame. Readers turn those
messages back into standard IPC before arrow-ipc decodes them, and pass
every other message through unchanged.

The rewritten record-batch messages use a BodyCompression codec Arrow
does not define, so readers that don't know the format fail instead of
misreading the data. `ballista.shuffle.compression.offsets_as_lengths`
(default true) turns it off.

Closes apache#2804

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@andygrove
andygrove force-pushed the perf/offsets-as-lengths branch from b156ad2 to fbdf495 Compare November 22, 2026 17:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api-change documentation Improvements or additions to documentation performance proto-change Changes the wire or serialization format

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Store shuffle string offsets as lengths before zstd compression

1 participant