Repository navigation
Conversation
…counts Partition coalescing skipped every stage whose shuffles feed an operator outside Ballista's hand-kept list, so an embedder that runs each stage as an operator of its own, such as Comet Native Executors, never got it. `AdaptiveRules` gains two methods with defaults that keep today's behavior. `tolerates_partition_count_change` lets the embedder accept an operator of its own, which the coalesce rule asks after the built-in list. `with_coalesced_children` lets it rebuild such an operator when the adapter turns the stage into the plan its tasks run, for an operator that keeps per-partition state beside its inputs. The scheduler now passes its physical codec and the embedder's rules to the adaptive planner together, and the planner keeps the rules for the coalesce rule and the adapter. Closes apache#2728 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
andygrove
force-pushed
the
andygrove/coalesce-tolerant-operators
branch
from
November 22, 2026 19:27
19327dc to
49f1fb6
Compare
andygrove
marked this pull request as ready for review
November 22, 2026 20:05
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #2728.
Rationale for this change
Partition coalescing (
ballista.planner.coalesce.enabled) only coalesces a stage when every operator between the stage root and its shuffles is on a hand-kept list of operators that produce the same rows over any partitioning of their input. Any other operator is assumed to depend on the partition count, which is the safe default. An embedder that runs each stage as an operator of its own never gets coalescing. Comet Native Executors runs every stage as aCometFragmentExec, so none of its queries are coalesced.A yes/no answer is not enough for every operator.
CometFragmentExechas a partition descriptor child with one row per partition, and it requires its shuffle inputs to have as many partitions as the descriptor. After coalescing, its shuffle readers have fewer partitions while the descriptor still has the old count, so rebuilding it withwith_new_childrenfails. Only the embedder knows how to rebuild that state.What changes are included in this PR?
AdaptiveRules, the existing extension point for an embedder's adaptive planning rules, gains two methods. Their defaults keep today's behavior.tolerates_partition_count_change(plan)is asked about operators that are not on Ballista's list. Returningtruelets the coalesce rule coalesce the shuffles below the operator.with_coalesced_children(plan, children, coalesce)is called when the adapter turns the stage into the plan its tasks run, for an operator the embedder accepted that sits above a coalesced shuffle. It gets theCoalescePlan, so it can rebuild per-partition state for the new count.Nonefalls back towith_new_children.CoalescePartitionsRuleand to the adapter.TaskManagerpasses the scheduler's physical codec (added in fix(scheduler): match reused exchanges with the scheduler's physical codec #2912) and the embedder's rules to the adaptive execution graph together in one internal struct, instead of as separate arguments.The new tests in
state/aqe/test/physical_submission.rs:should_coalesce_shuffle_feeding_tolerated_extension_operator: an extension operator the rules accept reads a coalesced shuffle. The existingshould_not_coalesce_shuffle_feeding_extension_operatorcovers the default.should_rebuild_extension_operator_over_coalesced_shuffle: a test operator shaped likeCometFragmentExecis rebuilt with a descriptor for the coalesced count.should_fail_extension_operator_its_rules_do_not_rebuild: the same operator fails to build when the rules accept it but do not rebuild it, which is why the hook is needed.Are there any user-facing changes?
Scheduler embedders can let partition coalescing apply to stages with operators of their own.
AdaptiveRuleshas two new methods with default implementations, so existing implementations keep compiling and behave as before. No breaking API changes.🤖 Generated with Claude Code