Skip to content

Charge native Parquet scan memory to the task memory pool #7688

Open

Description

@andygrove

What is the problem the feature request solves?

The native Parquet scan registers no memory reservation, so its working set is invisible to the task memory pool and to Spark behind it. #6125 prototyped charging it and was closed after review. This issue records what the prototype showed and what review found, so a later attempt does not start from scratch.

The gap. With tracing on (jemalloc,alloc-accounting), a TPC-H SF100 run (2 executors x 8 cores, 16g off-heap) showed scan-only queries such as q1, q6 and q19 allocating 100 to 150 MiB per executor natively while reserving almost nothing from the pool. The gap is a fixed working set per running scan, not something that grows over time. Per partition it is the compressed column chunks of the row group being decoded (arrow-rs fetches the projected column chunks of one row group before decoding it, and DataFusion reads one file per partition at a time), plus the decompressed pages and output batch the decoder builds. For q1 on SF100 lineitem the seven projected columns are 9.1 MiB compressed per row group, which matches the roughly 14 MiB per task that was unaccounted. The size is set by the file layout (row-group size times projected columns), so wide tables with large row groups can hold far more per task.

Describe the potential solution

The prototype is on the branch parquet-scan-memory-accounting of andygrove/datafusion-comet (head 3cb14c0f95). All of it is in EagerPageIndexReaderFactory, which already owns the scan's AsyncFileReader:

  • Data pages, charged exactly. Each buffer returned by get_byte_ranges is wrapped in a Bytes::from_owner whose owner holds its share of a MemoryReservation. The reservation is released when arrow-rs drops the last reference to the buffer or to any page sliced from it, so the decoder needs no change.
  • Decode buffers, charged as an estimate. arrow-rs allocates decompressed pages and the output batch with no hook to observe them, so each open reader held one default page (1 MiB) per projected leaf column plus one batch_size output batch.
  • Refusal does not fail the scan. The prototype used try_grow and kept reading unaccounted when refused, counting the bytes in a new scan_io_unreserved_bytes metric.

Measured results (TPC-H SF100, same commit with and without the change, native allocated bytes minus the pool's reserved total at the same instant, per executor):

Median gap p95 gap
Before 21.9 MiB 140.2 MiB
After 7.9 MiB 94.0 MiB

For scan-heavy queries the median gap dropped from 71 to 127 MiB to 6 to 19 MiB. Total run time was unchanged (208.7 s vs 208.8 s) and all 22 result hashes matched. At reduced off-heap size (fair_unified), no reservation was refused even at 2g, so the refusal path was not exercised on this workload. Spark's disk spill rose at 2g, mostly in q9 (8.0 to 15.0 GiB) and q10 (5.6 to 6.9 GiB). That is consistent with operators seeing less free memory, but it was a single run, so it needs repeating.

What review found

These need to be resolved by any follow-up. Thanks to @comphead, @viirya, @sunchao, @parthchandra, @rich7420 and @dwsmith1983 for the detailed reviews.

Error handling

  • try_grow can fail for reasons other than refusal. CometFairMemoryPool::try_grow returns ResourcesExhausted only for its own local-limit refusal and for a partial grant. Anything else (a SparkOutOfMemoryError from another consumer's spill, an InterruptedException from a killed task, a JNI failure) arrives as DataFusionError::External. The prototype used is_err(), which swallowed those, so a killed task kept reading. Only ResourcesExhausted should mean "keep reading unaccounted". Everything else must propagate, with a test using a pool whose try_grow returns External.
  • grow instead of try_grow is worth reconsidering. Since fix: let Comet memory pools overcommit on grow instead of panicking #6128 the unified pools no longer panic on grow. They record the bytes and carry what Spark declined as overcommit. The fetched pages already exist when charged, so this is not admission control. With grow the pressure lands on consumers that can spill, and both the error-classification question and scan_io_unreserved_bytes go away. The cost is that grow ignores the fair limit.
  • If try_grow stays, a scan that cannot act on a refusal keeps asking Spark for the whole overcommit debt on every call: a JNI round trip per row group, and possibly a JVM spill that buys nothing (Spark grants part, the unified pool hands the whole grant back, the scan reads anyway). Under greedy_unified each refusal also logs a warning and dumps all consumers' memory usage. The reader should back off after a refusal.

Consumer registration

  • One MemoryConsumer per reader shrinks the fair pool for the whole task. The fair pool's limit is pool_size / num, so each registration lowers every other consumer's ceiling regardless of how many bytes the scan holds (roughly 25% in an aggregate + sort + shuffle-writer task). DataFusion also creates a second reader per file for bloom-filter pruning and keeps both alive across load_bloom_filters, so the count churns and the decode estimate is charged twice. Register once on the factory, one consumer per scan operator, and hand each reader a new_empty() handle. This is the same pattern feat: charge native write buffers to the task's memory pool #6247 used for the writers. This may also explain part of the q9/q10 spill increase.

Decode estimate

  • leaves * 1 MiB + batch_size * row_width is fixed at plan time and charged before any metadata is read. A 500-column projection charged about 508 MiB per open reader (532,480,000 bytes in one reproduction) even for a one-row or 1 KiB file, and a scan over many small files pays it on every open. Capping by file size does not bound it either, since decompressed pages can exceed the file.
  • Size it from the file instead. The factory already loads the footer and offset index, so per projected leaf it could use the smaller of one page and the largest selected column chunk's uncompressed size, plus min(batch_size, num_rows) * row_width. Charge it at the first get_byte_ranges rather than in create_reader, which keeps it out of the planning step and means a fully pruned file never pays it.
  • row_width summed top-level fields (a struct counted as a flat 32 bytes) while the page term counted leaves, so the batch term under-counts wide nested rows. The test covered only Struct, not List<Struct> or Map.

Blocking and threading

  • create_reader runs in ParquetMorselizer::prepare_open_file, which DataFusion documents as the CPU-only planning step. A JNI acquireMemory there can run other consumers' spills and park in lock.wait().
  • Plans with no JVM sources are spawned onto the shared Tokio runtime, so grows in reserve_buffers can park a runtime worker too. This is most likely at the start of a task, which is exactly when a scan makes its first request. A task-kill interrupt is not delivered to a worker thread. fix: return a native plan's memory before its Spark task ends #6261 since made a blocked acquire hand the worker's other tasks to another thread, but that thread still waits in Spark.

Release accounting

  • Charging per buffer gives N shrinks (one JNI release_memory per column chunk, each taking the pool mutex). With the page index and a row selection, parquet fetches one range per selected page, so it can be one shrink per page. One Arc<MemoryReservation> shared by every buffer of a fetch gives one grow and one shrink, released when the last buffer drops.
  • Coalesced object-store fetches are undercharged. get_ranges coalesces nearby ranges, so each returned Bytes is a slice of one merged allocation and any surviving slice pins the gap bytes too. "Charged exactly" holds only for the local path. scan_io_object_store_response_bytes_read already measures the difference.
  • The design depends on an invariant that must be documented in the code: no page buffer may end up aliased by an Arrow array that crosses FFI. If it did, the final shrink would run from the FFI release callback when the JVM closes the batch. CometFairMemoryPool::shrink and CometUnifiedMemoryPool::shrink panic when the JNI release fails, and a panic in an extern "C" callback aborts the executor. Comet requests Utf8, not Utf8View, so this does not fire today. ByteViewArrayDecoderPlain can alias the page bytes, so a Utf8View hint in the file's ARROW:schema is safe only because the cast to the Spark type copies.

Scope

  • get_bytes (bloom filters) and get_metadata (footer plus page index) stay unreserved, and load_bloom_filters materializes BloomFilterStatistics for every row group at once, which is not small on a wide table.
  • The change turned accounting on for every native Parquet scan with no switch. A config flag (default true) would give users a way out while the estimate is refined.
  • scan_io_unreserved_bytes was cumulative refused bytes, including the one-shot decode estimate, but rendered as a size in the SQL UI so it read as a live footprint (or a leak). Any metric of this kind should be named and documented as cumulative, or split.
  • docs/source/contributor-guide/memory_management.md lists "decompression buffers, Parquet metadata structures, object_store request buffers" as undeclared. It, and its cgroup diagram, must be updated by whichever change closes the gap.
  • Spark SQL validation was not done. A change to scan memory behaviour needs the run-* Spark SQL label or dev/local-ci.sh spark sql_core-1, because the pull request tier runs only the default profile.

Suggested plan

  1. Register one MemoryConsumer per scan operator on the factory, with a new_empty() handle per reader.
  2. Share one Arc<MemoryReservation> per fetch for the data pages.
  3. Decide grow versus try_grow for the data pages. If try_grow, propagate non-ResourcesExhausted errors and back off after a refusal.
  4. Size the decode estimate from the file's metadata and charge it at the first fetch.
  5. Charge bloom-filter and metadata buffers, or document them as uncharged.
  6. Update memory_management.md, add a switch, and run Spark SQL validation.

Additional context

Closed attempt: #6125. Related: #6126 (the same gap in the native Iceberg scan, which cannot reuse the AsyncFileReader approach), #6127 and #6128 (pool grow behaviour).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:scanParquet scan / data readingenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions