Repository navigation
Conversation
A task that runs N partitions holds N vcores until its slowest partition ends. The shuffle writers now report each input partition they finish, and the executor hands that partition's vcore back while the task keeps running: in pull mode by releasing its semaphore permits, in push mode through a running task status that the scheduler refunds from the task's reservation. Controlled by ballista.scheduler.release_partition_vcores, on by default. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
andygrove
left a comment
There was a problem hiding this comment.
Thanks for this. The single-ledger design for the refunds is really nice, and it makes the push path easy to trust. A lost, late or repeated running report can't refund a vcore twice, and the abort paths only refund what the task still holds.
My main concern is memory, which is why I'm requesting changes. Details are inline.
Could you also share before/after numbers against main? CI shows the TPC queries still produce correct results, but TPC-H is mostly unskewed, so it doesn't say much about the win here or about the extra status traffic. Something like this at SF10, in both push and pull mode, would help:
cargo run --release --bin tpch -- benchmark ballista \
--host localhost --port 50050 \
--iterations 3 --path <tpch parquet dir> --format parquetA skewed workload, for example one large file next to many small ones, would show the benefit best.
| not apply to pipelined jobs, whose vcores are reserved per job." | ||
| .to_string(), | ||
| DataType::Boolean, | ||
| Some(true.to_string()), |
There was a problem hiding this comment.
My main concern is memory. The task's pool keeps the size it was given at launch after vcores are released. With FairSpillPool the straggler's fair share actually grows as the finished partitions unregister their consumers.
For example, take an 8-vcore executor with 32 GB, so 4 GB per vcore. An 8-partition task gets a 32 GB pool. Seven partitions finish, and the scheduler binds a 7-vcore task with a 28 GB pool on the released vcores. The straggler can now use the whole 32 GB, so the executor can be asked for about 60 GB. The executor sizes the pool from host memory by default, so this applies out of the box. Skew is also exactly when the straggler tends to be the memory-heavy partition, like a hot key in an aggregate or join.
Could the release also shrink the task's pool? One option is a pool wrapper with a lowerable limit that the releaser decrements by per_vcore * released. Another is to only release a vcore once its share of the pool is actually unreserved.
If that's a bigger change, would you be open to defaulting ballista.scheduler.release_partition_vcores to false until the pool follows the vcores?
| /// A task that reports released vcores while it runs still completes on | ||
| /// its terminal report, and gives back every vcore exactly once. | ||
| #[tokio::test] | ||
| async fn running_reports_release_vcores_and_leave_the_task_running() -> Result<()> { |
There was a problem hiding this comment.
I think this test would still pass if the running refunds were ignored, because the terminal status returns everything and the test only checks free == 4 at the end.
Could it block a task after its running report and assert that free vcores rise, or that a second task gets bound, before the terminal status arrives?
| ) | ||
| }; | ||
|
|
||
| // A pipelined task holds no vcores of its own to release. |
There was a problem hiding this comment.
Would it be worth a small test for the push side, like the pull-mode one in execution_loop.rs? It could check that a RunningTask status with vcores_released is sent, and that pipelined tasks and tasks with vcores_consumed <= 1 don't get a releaser.
| let rows = stats.num_rows.unwrap_or(0) as usize; | ||
| write_metrics.input_rows.add(rows); | ||
| write_metrics.output_rows.add(rows); | ||
| vcore_release::partition_completed(&context, num_partitions); |
There was a problem hiding this comment.
Question here. num_partitions is the child's output partition count. The comment above says coordinating operators like DynamicRangeRepartitionExec feed all K outputs from shared scatter tasks. In that case, does one output reaching EOF actually free a thread? Or can the scatter side still be busy on all inputs?
K also may not match the vcores the task was charged, so I wanted to check the assumption holds for that shape.
| /// Vcores the task's finished partitions no longer need, released since its | ||
| /// previous status. The task keeps the rest until its terminal status. | ||
| #[prost(uint32, tag = "2")] | ||
| pub vcores_released: u32, |
There was a problem hiding this comment.
Small one. The wire change is additive, but the new field breaks downstream struct literals and exhaustive patterns like RunningTask { executor_id }, the same way the in-tree call sites needed updating. Maybe worth a one-line note in docs/source/upgrading/55.0.0.md for library users?
| }) | ||
| .filter_map(|status| { | ||
| self.outstanding_task_reservations.remove(&( | ||
| let reservation = ( |
There was a problem hiding this comment.
Just to confirm my reading. Since the key includes the attempt number, a late running report from an old attempt can't refund vcores from a new one. That relies on task ids never being reused within a stage attempt. Is that guaranteed?
Which issue does this PR close?
Closes #2905.
Rationale for this change
A task that packs N partitions is charged N vcores and holds all of them until its slowest partition ends. When partitions are skewed (one large file or one hot key next to many small ones), most of those vcores sit idle for most of the task, while other stages and jobs wait for an executor with free vcores.
This PR hands a partition's vcore back as soon as that partition finishes, so the scheduler can bind new work to it while the rest of the task keeps running.
What changes are included in this PR?
PartitionVcoreReleaser(ballista/core/src/execution_plans/vcore_release.rs): attached to a task's session config by the executor. A task needsmin(running partitions, vcores)vcores and keeps at least one until it ends; the rest is released as partitions complete.ShuffleWriterExec,SortShuffleWriterExec,RangeShuffleWriterExec) report each input partition they finish draining.poll_workreports them as free. No protocol change is needed.RunningTaskstatus carrying the newvcores_releasedfield. The scheduler refunds those vcores from the task's entry inoutstanding_task_reservationsand leaves at least one for the terminal status, which takes what is left. Because every refund goes through that one ledger, a lost, late or repeated running report cannot refund a vcore twice, and abort and launch-failure paths refund only what the task still holds. Running reports are dropped before they reach the execution graph.ballista.scheduler.release_partition_vcores(defaulttrue) turns it off per session.Collapse stages hold one vcore and pipelined jobs reserve vcores per job, so neither releases anything.
Things worth a reviewer's attention:
BALLISTA_PROTOCOL_VERSION, which is already at 4 onmainand unreleased, so I did not bump it again.Are there any user-facing changes?
A new session setting,
ballista.scheduler.release_partition_vcores, on by default.RunningTaskgains avcores_releasedfield (proto-change).🤖 Generated with Claude Code