Skip to content

fix: allow standalone scheduler to use a session-configured partition slicing resolver - #2886

Merged
andygrove merged 2 commits into
apache:mainfrom
vikrantmehta123:fix/2839-standalone-partition-resolver
Nov 20, 2026
Merged

andygrove merged 2 commits into
apache:mainfrom
vikrantmehta123:fix/2839-standalone-partition-resolver

Conversation

@vikrantmehta123

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #2839 .

Rationale for this change

Standalone setup creates its own SchedulerConfig, leaving no way to supply a PartitionSliceableResolver. Custom plan nodes that require partition slicing therefore cannot run when their stages are split between tasks. The issue suggested two options. I have chosen the option of registering resolver next to the codec.

What changes are included in this PR?

  • Add resolver setter and getter methods to SessionConfigExt, following the existing codec extension pattern.
  • Forward the registered resolver to SchedulerConfig during standalone setup.
  • Add an integration test verifying that the standalone scheduler consults a resolver registered through session configuration.
  • Update the extending-components guide and upgrade notes.

Note: AI helped me identify that there are docs to be changed. AI also helped me come up with the integration test. I have read and understood both these changes, and they looked reasonable to me.

Are there any user-facing changes?

Yes, users can optionally register a resolver in standalone mode. The SessionConfigExt gets two more methods. This is documented in the upgrade guide.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @vikrantmehta123, this looks good. Reading the resolver from the session config next to the codec is a clean fit, and the upgrade note is clear. I added the api-change label since the note calls out the two new SessionConfigExt methods.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few optional follow-ups on top of my approval. None of these need to block the merge.

One more idea that sits outside the diff. The "cannot restrict" errors in ballista/scheduler/src/state/task_builder.rs still only point at SchedulerConfig::partition_sliceable_resolver. Now that standalone users have with_ballista_partition_sliceable_resolver, would it be worth mentioning it there too? That way the hint works in both modes.


// Execute a query to verify that the standalone scheduler uses the
// resolver registered in the session configuration.
ctx.sql(ALLTYPES_GROUP_BY_SQL).await?.collect().await?;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test is a nice, direct check that the resolver is consulted. Could it also compare the collected batches with ALLTYPES_GROUP_BY? That would show that a resolver returning None leaves results unchanged.

/// Returns the resolver for restricting custom plan nodes to a task's partitions.
///
/// Returns `None` if not set.
fn ballista_partition_sliceable_resolver(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional thought. ballista_partition_sliceable_resolver could default to None so outside implementors of SessionConfigExt only have to add the setter. Totally fine as is given the upgrade note.

@andygrove
andygrove added this pull request to the merge queue Nov 20, 2026
Merged via the queue into apache:main with commit d49c3bf Nov 20, 2026
25 checks passed
@vikrantmehta123

Copy link
Copy Markdown
Contributor Author

Have handled the comments in #2907.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api-change documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Let the standalone scheduler take a PartitionSliceableResolver

2 participants