Skip to content

feat: release a task's vcores as its partitions finish - #2904

Open
Dandandan wants to merge 2 commits into
apache:mainfrom
Dandandan:feat/release-partition-slots
Open

Dandandan wants to merge 2 commits into
apache:mainfrom
Dandandan:feat/release-partition-slots

Conversation

@Dandandan

@Dandandan Dandandan commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #2905.

Rationale for this change

A task that packs N partitions is charged N vcores and holds all of them until its slowest partition ends. When partitions are skewed (one large file or one hot key next to many small ones), most of those vcores sit idle for most of the task, while other stages and jobs wait for an executor with free vcores.

This PR hands a partition's vcore back as soon as that partition finishes, so the scheduler can bind new work to it while the rest of the task keeps running.

What changes are included in this PR?

  • PartitionVcoreReleaser (ballista/core/src/execution_plans/vcore_release.rs): attached to a task's session config by the executor. A task needs min(running partitions, vcores) vcores and keeps at least one until it ends; the rest is released as partitions complete.
  • Shuffle writers (ShuffleWriterExec, SortShuffleWriterExec, RangeShuffleWriterExec) report each input partition they finish draining.
  • Pull mode: the task splits the finished partitions' permits off its semaphore permit, so the next poll_work reports them as free. No protocol change is needed.
  • Push mode: the executor sends a RunningTask status carrying the new vcores_released field. The scheduler refunds those vcores from the task's entry in outstanding_task_reservations and leaves at least one for the terminal status, which takes what is left. Because every refund goes through that one ledger, a lost, late or repeated running report cannot refund a vcore twice, and abort and launch-failure paths refund only what the task still holds. Running reports are dropped before they reach the execution graph.
  • ballista.scheduler.release_partition_vcores (default true) turns it off per session.

Collapse stages hold one vcore and pipelined jobs reserve vcores per job, so neither releases anything.

Things worth a reviewer's attention:

  • Memory: a task's memory pool is sized by the vcores it was charged and is not shrunk when vcores are released. While a task with released vcores is still running, the task pools on an executor can add up to more than its memory. Finished partitions have stopped allocating, but the remaining ones can still use the whole pool.
  • Status traffic in push mode: one running status per released vcore (batched by the existing status reporter).
  • Executors only send running reports to a scheduler of the same BALLISTA_PROTOCOL_VERSION, which is already at 4 on main and unreleased, so I did not bump it again.

Are there any user-facing changes?

A new session setting, ballista.scheduler.release_partition_vcores, on by default. RunningTask gains a vcores_released field (proto-change).

🤖 Generated with Claude Code

A task that runs N partitions holds N vcores until its slowest partition
ends. The shuffle writers now report each input partition they finish, and
the executor hands that partition's vcore back while the task keeps running:
in pull mode by releasing its semaphore permits, in push mode through a
running task status that the scheduler refunds from the task's reservation.

Controlled by ballista.scheduler.release_partition_vcores, on by default.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@Dandandan Dandandan added run-tpch run-tpcds proto-change Changes the wire or serialization format labels Nov 20, 2026
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@Dandandan
Dandandan marked this pull request as ready for review November 20, 2026 15:11
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Nov 20, 2026

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. The single-ledger design for the refunds is really nice, and it makes the push path easy to trust. A lost, late or repeated running report can't refund a vcore twice, and the abort paths only refund what the task still holds.

My main concern is memory, which is why I'm requesting changes. Details are inline.

Could you also share before/after numbers against main? CI shows the TPC queries still produce correct results, but TPC-H is mostly unskewed, so it doesn't say much about the win here or about the extra status traffic. Something like this at SF10, in both push and pull mode, would help:

cargo run --release --bin tpch -- benchmark ballista \
  --host localhost --port 50050 \
  --iterations 3 --path <tpch parquet dir> --format parquet

A skewed workload, for example one large file next to many small ones, would show the benefit best.

not apply to pipelined jobs, whose vcores are reserved per job."
.to_string(),
DataType::Boolean,
Some(true.to_string()),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My main concern is memory. The task's pool keeps the size it was given at launch after vcores are released. With FairSpillPool the straggler's fair share actually grows as the finished partitions unregister their consumers.

For example, take an 8-vcore executor with 32 GB, so 4 GB per vcore. An 8-partition task gets a 32 GB pool. Seven partitions finish, and the scheduler binds a 7-vcore task with a 28 GB pool on the released vcores. The straggler can now use the whole 32 GB, so the executor can be asked for about 60 GB. The executor sizes the pool from host memory by default, so this applies out of the box. Skew is also exactly when the straggler tends to be the memory-heavy partition, like a hot key in an aggregate or join.

Could the release also shrink the task's pool? One option is a pool wrapper with a lowerable limit that the releaser decrements by per_vcore * released. Another is to only release a vcore once its share of the pool is actually unreserved.

If that's a bigger change, would you be open to defaulting ballista.scheduler.release_partition_vcores to false until the pool follows the vcores?

/// A task that reports released vcores while it runs still completes on
/// its terminal report, and gives back every vcore exactly once.
#[tokio::test]
async fn running_reports_release_vcores_and_leave_the_task_running() -> Result<()> {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this test would still pass if the running refunds were ignored, because the terminal status returns everything and the test only checks free == 4 at the end.

Could it block a task after its running report and assert that free vcores rise, or that a second task gets bound, before the terminal status arrives?

)
};

// A pipelined task holds no vcores of its own to release.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be worth a small test for the push side, like the pull-mode one in execution_loop.rs? It could check that a RunningTask status with vcores_released is sent, and that pipelined tasks and tasks with vcores_consumed <= 1 don't get a releaser.

let rows = stats.num_rows.unwrap_or(0) as usize;
write_metrics.input_rows.add(rows);
write_metrics.output_rows.add(rows);
vcore_release::partition_completed(&context, num_partitions);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question here. num_partitions is the child's output partition count. The comment above says coordinating operators like DynamicRangeRepartitionExec feed all K outputs from shared scatter tasks. In that case, does one output reaching EOF actually free a thread? Or can the scatter side still be busy on all inputs?

K also may not match the vcores the task was charged, so I wanted to check the assumption holds for that shape.

/// Vcores the task's finished partitions no longer need, released since its
/// previous status. The task keeps the rest until its terminal status.
#[prost(uint32, tag = "2")]
pub vcores_released: u32,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small one. The wire change is additive, but the new field breaks downstream struct literals and exhaustive patterns like RunningTask { executor_id }, the same way the in-tree call sites needed updating. Maybe worth a one-line note in docs/source/upgrading/55.0.0.md for library users?

})
.filter_map(|status| {
self.outstanding_task_reservations.remove(&(
let reservation = (

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to confirm my reading. Since the key includes the attempt number, a late running report from an old attempt can't refund vcores from a new one. That relies on task ids never being reused within a stage attempt. Is that guaranteed?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation proto-change Changes the wire or serialization format run-tpcds run-tpch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Release a task's vcores as its partitions finish

2 participants