You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Comet memory pools panic on MemoryPool::grow instead of allowing overcommit #6127
CometUnifiedMemoryPool and CometFairMemoryPool implement MemoryPool::grow as self.try_grow(..).unwrap() (unified_pool.rs:111, fair_pool.rs:121), so the task panics whenever Spark grants less than the full request. DataFusion documents grow as "Infallibly grow the provided reservation by additional bytes. This must always succeed", and its own pools implement it by adding to the counter and allowing the limit to be exceeded. Callers use grow, not try_grow, when the memory already exists and only needs to be recorded, so they have no way to handle a refusal. With Comet's pools the task dies and Spark retries it.
The panic was a deliberate trade-off. #1733 found that the old grow ignored a partial grant from Spark, so shrink could release more than was acquired. Of the two fixes proposed there, #1732 (panic when the grant falls short) was merged instead of #1731 (track what was actually acquired). That fixed the over-release but broke the grow contract.
Steps to reproduce
TPC-H SF100 on Spark 4.1.1, 2 executors x 8 cores, spark.memory.offHeap.size=2g, default fair_unified pool. Three tasks in one stage fail with:
Comet native panic: panicked at core/src/execution/memory_pools/fair_pool.rs:122:49:
called `Result::unwrap()` on an `Err` value: ResourcesExhausted("Failed to acquire 327992 bytes, only got 283353 bytes. Reserved: 357630588 bytes")
at datafusion_execution::memory_pool::MemoryReservation::grow
at datafusion_physical_plan::joins::sort_merge_join::materializing_stream::MaterializingSortMergeJoinStream::restore_spilled_batches
The sort-merge join calls grow for a spilled batch it has just read back from disk. That memory already exists, so there is nothing to refuse.
Expected behavior
grow never fails:
It acquires what Spark will grant.
It records any shortfall as overcommit, so reserved() still reflects the full amount in use.
Activity
fix: let Comet memory pools overcommit on grow instead of panicking (a…
fix: let Comet memory pools overcommit on grow instead of panicking (#…