Repository navigation
Description
What is the problem the feature request solves?
The native Parquet scan registers no memory reservation, so its working set is invisible to the task memory pool and to Spark behind it. #6125 prototyped charging it and was closed after review. This issue records what the prototype showed and what review found, so a later attempt does not start from scratch.
The gap. With tracing on (jemalloc,alloc-accounting), a TPC-H SF100 run (2 executors x 8 cores, 16g off-heap) showed scan-only queries such as q1, q6 and q19 allocating 100 to 150 MiB per executor natively while reserving almost nothing from the pool. The gap is a fixed working set per running scan, not something that grows over time. Per partition it is the compressed column chunks of the row group being decoded (arrow-rs fetches the projected column chunks of one row group before decoding it, and DataFusion reads one file per partition at a time), plus the decompressed pages and output batch the decoder builds. For q1 on SF100 lineitem the seven projected columns are 9.1 MiB compressed per row group, which matches the roughly 14 MiB per task that was unaccounted. The size is set by the file layout (row-group size times projected columns), so wide tables with large row groups can hold far more per task.
Describe the potential solution
The prototype is on the branch parquet-scan-memory-accounting of andygrove/datafusion-comet (head 3cb14c0f95). All of it is in EagerPageIndexReaderFactory, which already owns the scan's AsyncFileReader:
- Data pages, charged exactly. Each buffer returned by
get_byte_rangesis wrapped in aBytes::from_ownerwhose owner holds its share of aMemoryReservation. The reservation is released when arrow-rs drops the last reference to the buffer or to any page sliced from it, so the decoder needs no change. - Decode buffers, charged as an estimate. arrow-rs allocates decompressed pages and the output batch with no hook to observe them, so each open reader held one default page (1 MiB) per projected leaf column plus one
batch_sizeoutput batch. - Refusal does not fail the scan. The prototype used
try_growand kept reading unaccounted when refused, counting the bytes in a newscan_io_unreserved_bytesmetric.
Measured results (TPC-H SF100, same commit with and without the change, native allocated bytes minus the pool's reserved total at the same instant, per executor):
| Median gap | p95 gap | |
|---|---|---|
| Before | 21.9 MiB | 140.2 MiB |
| After | 7.9 MiB | 94.0 MiB |
For scan-heavy queries the median gap dropped from 71 to 127 MiB to 6 to 19 MiB. Total run time was unchanged (208.7 s vs 208.8 s) and all 22 result hashes matched. At reduced off-heap size (fair_unified), no reservation was refused even at 2g, so the refusal path was not exercised on this workload. Spark's disk spill rose at 2g, mostly in q9 (8.0 to 15.0 GiB) and q10 (5.6 to 6.9 GiB). That is consistent with operators seeing less free memory, but it was a single run, so it needs repeating.
What review found
These need to be resolved by any follow-up. Thanks to @comphead, @viirya, @sunchao, @parthchandra, @rich7420 and @dwsmith1983 for the detailed reviews.
Error handling
try_growcan fail for reasons other than refusal.CometFairMemoryPool::try_growreturnsResourcesExhaustedonly for its own local-limit refusal and for a partial grant. Anything else (aSparkOutOfMemoryErrorfrom another consumer's spill, anInterruptedExceptionfrom a killed task, a JNI failure) arrives asDataFusionError::External. The prototype usedis_err(), which swallowed those, so a killed task kept reading. OnlyResourcesExhaustedshould mean "keep reading unaccounted". Everything else must propagate, with a test using a pool whosetry_growreturnsExternal.growinstead oftry_growis worth reconsidering. Since fix: let Comet memory pools overcommit on grow instead of panicking #6128 the unified pools no longer panic ongrow. They record the bytes and carry what Spark declined as overcommit. The fetched pages already exist when charged, so this is not admission control. Withgrowthe pressure lands on consumers that can spill, and both the error-classification question andscan_io_unreserved_bytesgo away. The cost is thatgrowignores the fair limit.- If
try_growstays, a scan that cannot act on a refusal keeps asking Spark for the whole overcommit debt on every call: a JNI round trip per row group, and possibly a JVM spill that buys nothing (Spark grants part, the unified pool hands the whole grant back, the scan reads anyway). Undergreedy_unifiedeach refusal also logs a warning and dumps all consumers' memory usage. The reader should back off after a refusal.
Consumer registration
- One
MemoryConsumerper reader shrinks the fair pool for the whole task. The fair pool's limit ispool_size / num, so each registration lowers every other consumer's ceiling regardless of how many bytes the scan holds (roughly 25% in an aggregate + sort + shuffle-writer task). DataFusion also creates a second reader per file for bloom-filter pruning and keeps both alive acrossload_bloom_filters, so the count churns and the decode estimate is charged twice. Register once on the factory, one consumer per scan operator, and hand each reader anew_empty()handle. This is the same pattern feat: charge native write buffers to the task's memory pool #6247 used for the writers. This may also explain part of the q9/q10 spill increase.
Decode estimate
leaves * 1 MiB + batch_size * row_widthis fixed at plan time and charged before any metadata is read. A 500-column projection charged about 508 MiB per open reader (532,480,000 bytes in one reproduction) even for a one-row or 1 KiB file, and a scan over many small files pays it on every open. Capping by file size does not bound it either, since decompressed pages can exceed the file.- Size it from the file instead. The factory already loads the footer and offset index, so per projected leaf it could use the smaller of one page and the largest selected column chunk's uncompressed size, plus
min(batch_size, num_rows) * row_width. Charge it at the firstget_byte_rangesrather than increate_reader, which keeps it out of the planning step and means a fully pruned file never pays it. row_widthsummed top-level fields (a struct counted as a flat 32 bytes) while the page term counted leaves, so the batch term under-counts wide nested rows. The test covered onlyStruct, notList<Struct>orMap.
Blocking and threading
create_readerruns inParquetMorselizer::prepare_open_file, which DataFusion documents as the CPU-only planning step. A JNIacquireMemorythere can run other consumers' spills and park inlock.wait().- Plans with no JVM sources are spawned onto the shared Tokio runtime, so grows in
reserve_bufferscan park a runtime worker too. This is most likely at the start of a task, which is exactly when a scan makes its first request. A task-kill interrupt is not delivered to a worker thread. fix: return a native plan's memory before its Spark task ends #6261 since made a blocked acquire hand the worker's other tasks to another thread, but that thread still waits in Spark.
Release accounting
- Charging per buffer gives N shrinks (one JNI
release_memoryper column chunk, each taking the pool mutex). With the page index and a row selection, parquet fetches one range per selected page, so it can be one shrink per page. OneArc<MemoryReservation>shared by every buffer of a fetch gives one grow and one shrink, released when the last buffer drops. - Coalesced object-store fetches are undercharged.
get_rangescoalesces nearby ranges, so each returnedBytesis a slice of one merged allocation and any surviving slice pins the gap bytes too. "Charged exactly" holds only for the local path.scan_io_object_store_response_bytes_readalready measures the difference. - The design depends on an invariant that must be documented in the code: no page buffer may end up aliased by an Arrow array that crosses FFI. If it did, the final
shrinkwould run from the FFI release callback when the JVM closes the batch.CometFairMemoryPool::shrinkandCometUnifiedMemoryPool::shrinkpanic when the JNI release fails, and a panic in anextern "C"callback aborts the executor. Comet requestsUtf8, notUtf8View, so this does not fire today.ByteViewArrayDecoderPlaincan alias the page bytes, so aUtf8Viewhint in the file'sARROW:schemais safe only because the cast to the Spark type copies.
Scope
get_bytes(bloom filters) andget_metadata(footer plus page index) stay unreserved, andload_bloom_filtersmaterializesBloomFilterStatisticsfor every row group at once, which is not small on a wide table.- The change turned accounting on for every native Parquet scan with no switch. A config flag (default
true) would give users a way out while the estimate is refined. scan_io_unreserved_byteswas cumulative refused bytes, including the one-shot decode estimate, but rendered as a size in the SQL UI so it read as a live footprint (or a leak). Any metric of this kind should be named and documented as cumulative, or split.docs/source/contributor-guide/memory_management.mdlists "decompression buffers, Parquet metadata structures,object_storerequest buffers" as undeclared. It, and its cgroup diagram, must be updated by whichever change closes the gap.- Spark SQL validation was not done. A change to scan memory behaviour needs the
run-*Spark SQL label ordev/local-ci.sh spark sql_core-1, because the pull request tier runs only the default profile.
Suggested plan
- Register one
MemoryConsumerper scan operator on the factory, with anew_empty()handle per reader. - Share one
Arc<MemoryReservation>per fetch for the data pages. - Decide
growversustry_growfor the data pages. Iftry_grow, propagate non-ResourcesExhaustederrors and back off after a refusal. - Size the decode estimate from the file's metadata and charge it at the first fetch.
- Charge bloom-filter and metadata buffers, or document them as uncharged.
- Update
memory_management.md, add a switch, and run Spark SQL validation.
Additional context
Closed attempt: #6125. Related: #6126 (the same gap in the native Iceberg scan, which cannot reuse the AsyncFileReader approach), #6127 and #6128 (pool grow behaviour).
Activity