Skip to content

fix: [branch-1.1] do not resolve DPP subqueries when AQE reads a native scan's partitioning (#6268) - #7624

Draft
dwsmith1983 wants to merge 1 commit into
apache:branch-1.1from
dwsmith1983:fix/6268-backport-branch-1.1
Draft

dwsmith1983 wants to merge 1 commit into
apache:branch-1.1from
dwsmith1983:fix/6268-backport-branch-1.1

Conversation

@dwsmith1983

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Backport of #6268 to branch-1.1. Part of #6133.

#7623 must merge first: one of the tests here needs it. This PR and #7338 (the branch-1.1 backport of #6270, the other half of #6133) don't depend on each other; whichever merges second should take branch-1.1 and rerun its suites. branch-1.2 already has #6268. branch-1.0 has the same code; no backport is planned there.

Rationale for this change

AQE reads outputPartitioning when CoalesceShufflePartitions changes a stage, before the DPP placeholders are converted. On branch-1.1, a non-bucketed CometNativeScanExec answers with its split count, which runs the placeholder and fails with SubqueryAdaptiveBroadcastExec does not support the execute() code path. CometIcebergNativeScanExec fails the same way.

What changes are included in this PR?

Cherry-pick of 695ca3a with -x.

  • A non-bucketed native scan and the Iceberg scan report UnknownPartitioning(0). A bucketed scan keeps its bucket layout.
  • CometNativeExec sizes their native RDD from the plan data at execution.

Adaptations: the CometTaskMetricsSuite change is dropped, because the test it edits is not on this branch, and two suites import what the new tests use, which main already had.

How are these changes tested?

The source PR's three tests. In a Linux container on Spark 4.1, CometExecSuite and CometIcebergNativeSuite pass apart from the bucketed test, plus the scan, join and DPP suites. On Spark 3.5 the AQE DPP tests in both suites pass apart from the same test. With spark/src/main reverted, both shuffle coalescing tests fail with the errors above.

The bucketed test fails on branch-1.1 with or without this change, on an empty bucket's scan schema, which #7623 fixes. With both applied, all three tests pass on Spark 4.1 and 3.5.

…ve scan's partitioning (apache#6268)

AQE validates a stage by reading outputPartitioning after
CoalesceShufflePartitions changes it, which is before Comet converts the
DPP placeholders. CometNativeScanExec answered with its post-pruning
split count, which ran the SubqueryAdaptiveBroadcastExec placeholder and
threw, and CometIcebergNativeScanExec ran BatchScanExec.inputRDD on a DPP
subquery with no result yet. A non-bucketed native scan and the Iceberg
scan now report UnknownPartitioning(0), and CometNativeExec sizes the
native RDD from the scan's plan data at execution.

Adapted to branch-1.1:
- The CometTaskMetricsSuite change is left out: the test it edits is not
  on this branch.
- CometIcebergNativeSuite keeps this branch's imports and adds only
  AQEShuffleReadExec.
- CometExecSuite imports AQEShuffleReadExec and ShuffleExchangeLike,
  which main already imported before apache#6268.

Part of apache#6133.

(cherry picked from commit 695ca3a)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant