Skip to content

fix(scheduler): match reused exchanges with the scheduler's physical codec - #2912

Merged
andygrove merged 2 commits into
apache:mainfrom
andygrove:andygrove/reuse-exchange-scheduler-codec
Nov 22, 2026
Merged

andygrove merged 2 commits into
apache:mainfrom
andygrove:andygrove/reuse-exchange-scheduler-codec

Conversation

@andygrove

@andygrove andygrove commented Nov 22, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes #2908.

Rationale for this change

Exchange reuse (ballista.planner.reuse_exchange.enabled, on by default) never shares a stage for an embedder that registers its physical extension codec only through SchedulerConfig::override_physical_codec. The reuse key encodes each stage with the session's codec, which is Ballista's default unless the client also sets the codec on the session. The default codec cannot encode the embedder's operators, so the key is None and the stage runs again. Nothing is logged.

With Comet Native Executors on TPC-H SF100, q18 ran its lineitem partial aggregate twice. Setting the codec on the session as a workaround made the stages shared and took q18 from about 41s to 20s on one 16-slot executor.

What changes are included in this PR?

  • AdaptivePlanner keeps the codec it encodes reuse keys with. It defaults to the session's codec, and TaskManager replaces it with the scheduler's codec, the one stages ship to executors with. This matches the intent in the reuse_exchange module docs that two stages match only when an executor could not tell them apart.
  • AdaptiveExecutionGraph::try_new and try_new_from_physical take the codec as a required argument. The aqe module is crate-private, so this is not a public API change.
  • reuse_key logs at debug level when a stage cannot be encoded, so a stage that is silently not shared is visible.
  • BallistaCodec::physical_extension_codec_arc returns a shared handle to the physical codec, since the planner holds it for the life of the job.

The new test an_embedders_stages_are_matched_with_the_scheduler_codec plans two identical exchanges over an operator that only a test codec can encode. With the session's codec it gets two stages, as before this change. With the scheduler's codec it gets one.

Are there any user-facing changes?

Embedders that set SchedulerConfig::override_physical_codec now get exchange reuse without also setting the codec on the session. There is one additive public method, BallistaCodec::physical_extension_codec_arc. No breaking API changes.

🤖 Generated with Claude Code

sim and others added 2 commits November 22, 2026 08:11
…codec

Exchange reuse keyed each stage by its encoding under the session's
physical extension codec. An embedder that registers its codec only with
`SchedulerConfig::override_physical_codec` leaves the session with
Ballista's default codec, which cannot encode the embedder's operators,
so none of its stages were ever shared and nothing was logged.

The adaptive planner now encodes stages for the reuse key with the codec
the scheduler ships them with, and logs at debug level when a stage
cannot be encoded.

Closes apache#2908

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…etup

The adaptive graph constructors now take the scheduler's physical codec
as a required argument, since every production caller has one, which
drops the optional fallback and its helper. The reuse key logs encode
failures with `inspect_err`, and the reuse tests share one helper for
planners over a union of exchanges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@milenkovicm milenkovicm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove

@andygrove
andygrove added this pull request to the merge queue Nov 22, 2026
@andygrove

Copy link
Copy Markdown
Member Author

Thanks @milenkovicm

Merged via the queue into apache:main with commit 7faef66 Nov 22, 2026
24 checks passed
@andygrove
andygrove deleted the andygrove/reuse-exchange-scheduler-codec branch November 22, 2026 19:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Research: exchange reuse encodes stages with the session's codec, not the scheduler's override codec, so embedders' stages are never shared

2 participants