Repository navigation
fix(scheduler): match reused exchanges with the scheduler's physical codec - #2912
Merged
andygrove merged 2 commits intoNov 22, 2026
Merged
Conversation
…codec Exchange reuse keyed each stage by its encoding under the session's physical extension codec. An embedder that registers its codec only with `SchedulerConfig::override_physical_codec` leaves the session with Ballista's default codec, which cannot encode the embedder's operators, so none of its stages were ever shared and nothing was logged. The adaptive planner now encodes stages for the reuse key with the codec the scheduler ships them with, and logs at debug level when a stage cannot be encoded. Closes apache#2908 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…etup The adaptive graph constructors now take the scheduler's physical codec as a required argument, since every production caller has one, which drops the optional fallback and its helper. The reuse key logs encode failures with `inspect_err`, and the reuse tests share one helper for planners over a union of exchanges. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
|
Thanks @milenkovicm |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #2908.
Rationale for this change
Exchange reuse (
ballista.planner.reuse_exchange.enabled, on by default) never shares a stage for an embedder that registers its physical extension codec only throughSchedulerConfig::override_physical_codec. The reuse key encodes each stage with the session's codec, which is Ballista's default unless the client also sets the codec on the session. The default codec cannot encode the embedder's operators, so the key isNoneand the stage runs again. Nothing is logged.With Comet Native Executors on TPC-H SF100, q18 ran its
lineitempartial aggregate twice. Setting the codec on the session as a workaround made the stages shared and took q18 from about 41s to 20s on one 16-slot executor.What changes are included in this PR?
AdaptivePlannerkeeps the codec it encodes reuse keys with. It defaults to the session's codec, andTaskManagerreplaces it with the scheduler's codec, the one stages ship to executors with. This matches the intent in thereuse_exchangemodule docs that two stages match only when an executor could not tell them apart.AdaptiveExecutionGraph::try_newandtry_new_from_physicaltake the codec as a required argument. Theaqemodule is crate-private, so this is not a public API change.reuse_keylogs at debug level when a stage cannot be encoded, so a stage that is silently not shared is visible.BallistaCodec::physical_extension_codec_arcreturns a shared handle to the physical codec, since the planner holds it for the life of the job.The new test
an_embedders_stages_are_matched_with_the_scheduler_codecplans two identical exchanges over an operator that only a test codec can encode. With the session's codec it gets two stages, as before this change. With the scheduler's codec it gets one.Are there any user-facing changes?
Embedders that set
SchedulerConfig::override_physical_codecnow get exchange reuse without also setting the codec on the session. There is one additive public method,BallistaCodec::physical_extension_codec_arc. No breaking API changes.🤖 Generated with Claude Code