Skip to content

feat(scheduler): let embedder operators tolerate coalesced partition counts - #2913

Open
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:andygrove/coalesce-tolerant-operators
Open

andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:andygrove/coalesce-tolerant-operators

Conversation

@andygrove

@andygrove andygrove commented Nov 22, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes #2728.

Rationale for this change

Partition coalescing (ballista.planner.coalesce.enabled) only coalesces a stage when every operator between the stage root and its shuffles is on a hand-kept list of operators that produce the same rows over any partitioning of their input. Any other operator is assumed to depend on the partition count, which is the safe default. An embedder that runs each stage as an operator of its own never gets coalescing. Comet Native Executors runs every stage as a CometFragmentExec, so none of its queries are coalesced.

A yes/no answer is not enough for every operator. CometFragmentExec has a partition descriptor child with one row per partition, and it requires its shuffle inputs to have as many partitions as the descriptor. After coalescing, its shuffle readers have fewer partitions while the descriptor still has the old count, so rebuilding it with with_new_children fails. Only the embedder knows how to rebuild that state.

What changes are included in this PR?

  • AdaptiveRules, the existing extension point for an embedder's adaptive planning rules, gains two methods. Their defaults keep today's behavior.
    • tolerates_partition_count_change(plan) is asked about operators that are not on Ballista's list. Returning true lets the coalesce rule coalesce the shuffles below the operator.
    • with_coalesced_children(plan, children, coalesce) is called when the adapter turns the stage into the plan its tasks run, for an operator the embedder accepted that sits above a coalesced shuffle. It gets the CoalescePlan, so it can rebuild per-partition state for the new count. None falls back to with_new_children.
  • The adaptive planner keeps the embedder's rules and hands them to CoalescePartitionsRule and to the adapter.
  • TaskManager passes the scheduler's physical codec (added in fix(scheduler): match reused exchanges with the scheduler's physical codec #2912) and the embedder's rules to the adaptive execution graph together in one internal struct, instead of as separate arguments.
  • The architecture guide mentions the two methods next to its note that coalescing skips operators the rule does not know.

The new tests in state/aqe/test/physical_submission.rs:

  • should_coalesce_shuffle_feeding_tolerated_extension_operator: an extension operator the rules accept reads a coalesced shuffle. The existing should_not_coalesce_shuffle_feeding_extension_operator covers the default.
  • should_rebuild_extension_operator_over_coalesced_shuffle: a test operator shaped like CometFragmentExec is rebuilt with a descriptor for the coalesced count.
  • should_fail_extension_operator_its_rules_do_not_rebuild: the same operator fails to build when the rules accept it but do not rebuild it, which is why the hook is needed.

Are there any user-facing changes?

Scheduler embedders can let partition coalescing apply to stages with operators of their own. AdaptiveRules has two new methods with default implementations, so existing implementations keep compiling and behave as before. No breaking API changes.

🤖 Generated with Claude Code

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Nov 22, 2026
…counts

Partition coalescing skipped every stage whose shuffles feed an operator
outside Ballista's hand-kept list, so an embedder that runs each stage as
an operator of its own, such as Comet Native Executors, never got it.

`AdaptiveRules` gains two methods with defaults that keep today's
behavior. `tolerates_partition_count_change` lets the embedder accept an
operator of its own, which the coalesce rule asks after the built-in
list. `with_coalesced_children` lets it rebuild such an operator when
the adapter turns the stage into the plan its tasks run, for an operator
that keeps per-partition state beside its inputs.

The scheduler now passes its physical codec and the embedder's rules to
the adaptive planner together, and the planner keeps the rules for the
coalesce rule and the adapter.

Closes apache#2728

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@andygrove
andygrove force-pushed the andygrove/coalesce-tolerant-operators branch from 19327dc to 49f1fb6 Compare November 22, 2026 19:27
@andygrove
andygrove marked this pull request as ready for review November 22, 2026 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Let custom operators declare that they tolerate a coalesced partition count

1 participant