Repository navigation
fix: [branch-1.1] do not resolve DPP subqueries when AQE reads a native scan's partitioning (#6268) - #7624
Draft
dwsmith1983 wants to merge 1 commit into
Draft
fix: [branch-1.1] do not resolve DPP subqueries when AQE reads a native scan's partitioning (#6268)#7624dwsmith1983 wants to merge 1 commit into
dwsmith1983 wants to merge 1 commit into
Conversation
…ve scan's partitioning (apache#6268) AQE validates a stage by reading outputPartitioning after CoalesceShufflePartitions changes it, which is before Comet converts the DPP placeholders. CometNativeScanExec answered with its post-pruning split count, which ran the SubqueryAdaptiveBroadcastExec placeholder and threw, and CometIcebergNativeScanExec ran BatchScanExec.inputRDD on a DPP subquery with no result yet. A non-bucketed native scan and the Iceberg scan now report UnknownPartitioning(0), and CometNativeExec sizes the native RDD from the scan's plan data at execution. Adapted to branch-1.1: - The CometTaskMetricsSuite change is left out: the test it edits is not on this branch. - CometIcebergNativeSuite keeps this branch's imports and adds only AQEShuffleReadExec. - CometExecSuite imports AQEShuffleReadExec and ShuffleExchangeLike, which main already imported before apache#6268. Part of apache#6133. (cherry picked from commit 695ca3a)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Backport of #6268 to
branch-1.1. Part of #6133.#7623 must merge first: one of the tests here needs it. This PR and #7338 (the
branch-1.1backport of #6270, the other half of #6133) don't depend on each other; whichever merges second should takebranch-1.1and rerun its suites.branch-1.2already has #6268.branch-1.0has the same code; no backport is planned there.Rationale for this change
AQE reads
outputPartitioningwhenCoalesceShufflePartitionschanges a stage, before the DPP placeholders are converted. Onbranch-1.1, a non-bucketedCometNativeScanExecanswers with its split count, which runs the placeholder and fails withSubqueryAdaptiveBroadcastExec does not support the execute() code path.CometIcebergNativeScanExecfails the same way.What changes are included in this PR?
Cherry-pick of 695ca3a with
-x.UnknownPartitioning(0). A bucketed scan keeps its bucket layout.CometNativeExecsizes their native RDD from the plan data at execution.Adaptations: the
CometTaskMetricsSuitechange is dropped, because the test it edits is not on this branch, and two suites import what the new tests use, whichmainalready had.How are these changes tested?
The source PR's three tests. In a Linux container on Spark 4.1,
CometExecSuiteandCometIcebergNativeSuitepass apart from the bucketed test, plus the scan, join and DPP suites. On Spark 3.5 the AQE DPP tests in both suites pass apart from the same test. Withspark/src/mainreverted, both shuffle coalescing tests fail with the errors above.The bucketed test fails on
branch-1.1with or without this change, on an empty bucket's scan schema, which #7623 fixes. With both applied, all three tests pass on Spark 4.1 and 3.5.