Repository navigation
fix: allow standalone scheduler to use a session-configured partition slicing resolver - #2886
Conversation
andygrove
left a comment
There was a problem hiding this comment.
Thanks @vikrantmehta123, this looks good. Reading the resolver from the session config next to the codec is a clean fit, and the upgrade note is clear. I added the api-change label since the note calls out the two new SessionConfigExt methods.
andygrove
left a comment
There was a problem hiding this comment.
A few optional follow-ups on top of my approval. None of these need to block the merge.
One more idea that sits outside the diff. The "cannot restrict" errors in ballista/scheduler/src/state/task_builder.rs still only point at SchedulerConfig::partition_sliceable_resolver. Now that standalone users have with_ballista_partition_sliceable_resolver, would it be worth mentioning it there too? That way the hint works in both modes.
|
|
||
| // Execute a query to verify that the standalone scheduler uses the | ||
| // resolver registered in the session configuration. | ||
| ctx.sql(ALLTYPES_GROUP_BY_SQL).await?.collect().await?; |
There was a problem hiding this comment.
The test is a nice, direct check that the resolver is consulted. Could it also compare the collected batches with ALLTYPES_GROUP_BY? That would show that a resolver returning None leaves results unchanged.
| /// Returns the resolver for restricting custom plan nodes to a task's partitions. | ||
| /// | ||
| /// Returns `None` if not set. | ||
| fn ballista_partition_sliceable_resolver( |
There was a problem hiding this comment.
Optional thought. ballista_partition_sliceable_resolver could default to None so outside implementors of SessionConfigExt only have to add the setter. Totally fine as is given the upgrade note.
|
Have handled the comments in #2907. |
Which issue does this PR close?
Closes #2839 .
Rationale for this change
Standalone setup creates its own
SchedulerConfig, leaving no way to supply aPartitionSliceableResolver. Custom plan nodes that require partition slicing therefore cannot run when their stages are split between tasks. The issue suggested two options. I have chosen the option of registering resolver next to the codec.What changes are included in this PR?
SessionConfigExt, following the existing codec extension pattern.SchedulerConfigduring standalone setup.Note: AI helped me identify that there are docs to be changed. AI also helped me come up with the integration test. I have read and understood both these changes, and they looked reasonable to me.
Are there any user-facing changes?
Yes, users can optionally register a resolver in standalone mode. The
SessionConfigExtgets two more methods. This is documented in the upgrade guide.