Repository navigation
fix: keep a DPP build side's storage-partitioned join free of exchanges - #7662
snmvaughan wants to merge 4 commits into
Conversation
AQE plans a DPP subquery's converted build side again, and EnsureRequirements then puts an exchange under each side of a Comet join that Spark planned as a storage-partitioned join, because the join declares the original operator's required distribution since apache#7231. AQE has no logical link for those exchanges and asserts. Spark only accepts a key-grouped join as co-partitioned when the parent is its own SortMergeJoinExec or ShuffledHashJoinExec, and in Spark 4.1 when every leaf is also a KeyGroupedPartitionedScan. Reporting the scan's partitioning therefore cannot help a join. Declare no required distribution on CometSortMergeJoinExec and CometHashJoinExec when both sides are storage-partitioned. An aggregate above such a join does need the scan's partitioning, or the final aggregate gets an exchange inside an already serialized native block. So CometIcebergNativeScanExec now reports whatever the BatchScanExec it replaces reports, which generalizes apache#7424 from SinglePartition. Add a test for the join and one for an aggregate above it. Closes apache#7441.
mbutrovich
left a comment
There was a problem hiding this comment.
Thanks @snmvaughan for tracking this down, and for the clear explanation of why the scan change alone can't fix the join. #7510 brought #7231 to branch-1.2, so branch-1.2 has this regression too, and I'll add the backport-1.2 label.
| // Spark's own scans. Comet's non-native V2 scan, CometBatchScanExec, is a leaf under a | ||
| // CometScanWrapper that this does not unwrap, so a join over it keeps its requirement. |
There was a problem hiding this comment.
This comment describes a path that no scan takes today. CometBatchScanExec is only created for CSV, Text, and Iceberg (CometScanRule.scala L415, L574, L1345), and CometExecRule handles each of those in its own case before the CometScanWrapper one (CometExecRule.scala L470-L483). An Iceberg scan that the native conversion declines stays a bare CometBatchScanExec. That class extends DataSourceV2ScanExecBase, so it matches the case below this comment and isn't excluded.
Could we reword the comment to match? The paragraph in the PR description about Comet's non-native Iceberg reader has the same problem, so it could go too.
| // Spark's own scans. Comet's non-native V2 scan, CometBatchScanExec, is a leaf under a | |
| // CometScanWrapper that this does not unwrap, so a join over it keeps its requirement. | |
| // Spark's own scans, and CometBatchScanExec, which extends DataSourceV2ScanExecBase. |
There was a problem hiding this comment.
You're right, thank you. CometBatchScanExec is only built for CSV, Text and Iceberg, and CometExecRule handles each of them before the CometScanWrapper case, so a declined Iceberg conversion stays a bare CometBatchScanExec that this case already matches. I used your wording for the comment and removed the paragraph from the description. Done in 7835efe.
| val query = | ||
| """SELECT /*+ BROADCAST(d) */ f.store_id, f.amount, d.name | ||
| |FROM aqe_cat.db.spj_dpp_fact f JOIN ( | ||
| | SELECT /*+ MERGE(a, b) */ a.id, b.name |
There was a problem hiding this comment.
CometHashJoinExec gets the new trait too, but both tests use MERGE. Could we also run them with SHUFFLE_HASH(a, b) and assert that the plan has a CometHashJoinExec? I tried that variant of the join test locally on Spark 4.1. It fails with the same AssertionError in setLogicalLinkForNewQueryStage when operators.scala is reverted to main, and passes with this PR.
The two tests also share all of their setup, including the table names. When I ran them without the operators.scala change, the aggregate test failed with TABLE_OR_VIEW_ALREADY_EXISTS on spj_dpp_fact, because the join test failed before its DROP TABLE lines ran. So a failure in one test hides the result of the other. Could we use one setup with the drops in a finally, and generate the cases with a for over the two queries and the two hints, like the test at L6170-L6174? That would also give the aggregate query the no-exchange assertion that only the join query has now.
There was a problem hiding this comment.
Done in 7835efe. The two queries and the two hints are now generated with a for, the tables are created once per case and dropped in a finally, and every case asserts the join operator the hint should produce (CometSortMergeJoinExec for MERGE, CometHashJoinExec for SHUFFLE_HASH), that the fact scan carries a DPP filter, and that neither the plan nor the DPP subquery's plan has an exchange, which the aggregate query now gets too. Thank you for running the SHUFFLE_HASH variant against a reverted operators.scala.
Generate the join and the aggregate test for the MERGE and SHUFFLE_HASH hints, so CometSortMergeJoinExec and CometHashJoinExec are both covered. Each case asserts the join operator, that the fact scan carries a DPP filter, and that neither the plan nor the DPP subquery's plan has an exchange. The tables are created once per case and dropped in a finally, so a failing case no longer makes the next one fail on CREATE TABLE. Correct the comment in isStoragePartitioned. A declined Iceberg scan stays a bare CometBatchScanExec, which extends DataSourceV2ScanExecBase and is matched by that case.
comphead
left a comment
There was a problem hiding this comment.
I checked the EnsureRequirements reasoning against Spark 3.5.8, 4.0.1 and 4.1.1. checkKeyGroupCompatible only accepts Spark's SortMergeJoinExec or ShuffledHashJoinExec as the parent, so dropping the requirement on the Comet join is the change that can work. serializePartitions builds from inputRDD, which pads pruned key groups, so the reported partition count matches what runs.
| originalPlan.outputPartitioning match { | ||
| case _: UnknownPartitioning => UnknownPartitioning(0) | ||
| case reported => reported | ||
| } |
There was a problem hiding this comment.
When BatchScanExec reports no partitioning, it returns UnknownPartitioning(0) from SparkPlan.outputPartitioning on every version I checked (v3.4.3, v3.5.8, v4.0.1, v4.1.1, v4.2.1). So this match always returns its input. CometNativeScanExec.outputPartitioning returns originalPlan.outputPartitioning directly for a bucketed scan, and this could do the same:
| originalPlan.outputPartitioning match { | |
| case _: UnknownPartitioning => UnknownPartitioning(0) | |
| case reported => reported | |
| } | |
| originalPlan.outputPartitioning |
There was a problem hiding this comment.
Done in 6dab540, thank you. I checked DataSourceV2ScanExecBase.outputPartitioning on Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0, and its fallback is super.outputPartitioning in each, so the match was dead code. The originalPlan == null guard stays, because a canonicalized copy has no original plan.
…ed-dpp-join-exchanges # Conflicts: # spark/src/main/scala/org/apache/spark/sql/comet/CometIcebergNativeScanExec.scala
BatchScanExec reports UnknownPartitioning(0) when it has no partitioning, on every supported Spark version, so mapping an unknown partitioning to UnknownPartitioning(0) changed nothing. Return the replaced scan's outputPartitioning directly, as CometNativeScanExec does for a bucketed scan.
Which issue does this PR close?
Closes #7441.
Rationale for this change
With AQE on, a query fails with an
AssertionErrorinAdaptiveSparkPlanExec.setLogicalLinkForNewQueryStagewhen the build side of a DPP broadcast contains a storage-partitioned join over native Iceberg scans.Comet builds the DPP subquery's plan by wrapping the build side it already converted in a new
AdaptiveSparkPlanExec, and that constructor runsEnsureRequirementsover it again. Since #7231 the Comet joins, aggregates and windows declare Spark's required child distribution, andEnsureRequirementsputs an exchange under each side of the join. AQE has no logical link for those exchanges.The issue suggests having the native scan report the key-grouped partitioning of the scan it replaces, as #7424 did for
SinglePartition. That is not enough for a join. In Spark 3.5 through 4.2,EnsureRequirementsonly treats two key-grouped children as co-partitioned when the parent is Spark's ownSortMergeJoinExecorShuffledHashJoinExec(checkKeyGroupCompatible), and in 4.1 every leaf below them must also be aKeyGroupedPartitionedScan(canPushDownSPJParamsToScan). ACometSortMergeJoinExecoverCometIcebergNativeScanExecs is neither, so it finds no usable shuffle spec and shuffles both sides whatever the scans report. With only the scan change, the new join test fails in the same way.An aggregate has one child, so
EnsureRequirementsonly asks whether that child's partitioning satisfies the distribution. An aggregate above a storage-partitioned join does need the scans to report their partitioning. Without it the final aggregate gets an exchange between itself and the partial aggregate, inside a native block that was already serialized, and the native side fails withColumn index 1 is out of bound.What changes are included in this PR?
CometSortMergeJoinExecandCometHashJoinExecdeclare no required child distribution when both sides are storage-partitioned, meaning every leaf is a key-grouped scan and there is no shuffle above it. No shuffle sits below such a join for AQE to split, so the requirement from fix: declare required child distribution on aggregate, window and join operators #7231 protects nothing there. Every other join still returns the original operator's requirement.CometExec.isStoragePartitionedchecks the join's current children, because conversion works bottom-up andoriginalPlan.childrenare already Comet nodes. It usesDataSourceV2ScanExecBase.keyGroupedPartitioning, which every supported Spark version declares, so it needs no shim even though Spark 4.2 renamedKeyGroupedPartitioningtoKeyedPartitioning.CometIcebergNativeScanExec.outputPartitioningreports the partitioning of theBatchScanExecit replaces, instead of alwaysUnknownPartitioning(0), asCometNativeScanExecdoes for a bucketed scan. Reading it resolves no DPP subquery, because in Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0BatchScanExec.outputPartitioningis built frominputPartitionsand onlyinputRDDusesfilteredPartitions.CometIcebergNativeSuite, generated from two queries and two join hints. One query is the reproduction from the issue and the other puts an aggregate above the same storage-partitioned join. TheMERGEhint gives aCometSortMergeJoinExecandSHUFFLE_HASHaCometHashJoinExec. Each case asserts that the plan has the expected join operator, that the fact scan carries a DPP filter, and that neither the plan nor the DPP subquery's plan has an exchange. The tables are dropped in afinally, so a failing case does not make the next one fail onCREATE TABLE.How are these changes tested?
CometIcebergNativeSuitehas four cases for this: the join and the aggregate, each with theMERGEand theSHUFFLE_HASHhint. They were restructured during review, so CI is the first run of the four as they are now.Before that, on Spark 4.1:
Column index 1 is out of bound, and passed with it.AQE DPPtests inCometIcebergNativeSuitepassed.A reviewer ran the
SHUFFLE_HASHjoin case on Spark 4.1 againstmain'soperators.scala, where it fails with the sameAssertionError, and it passes with this PR.The change compiles on Spark 3.5 with Scala 2.12 and on Spark 4.1 with its tests, and its main sources compile on Spark 4.2. I have not run the tests on Spark 3.5 or 4.0.