Skip to content

fix: keep a DPP build side's storage-partitioned join free of exchanges - #7662

Open
snmvaughan wants to merge 4 commits into
apache:mainfrom
snmvaughan:fix/storage-partitioned-dpp-join-exchanges
Open

snmvaughan wants to merge 4 commits into
apache:mainfrom
snmvaughan:fix/storage-partitioned-dpp-join-exchanges

Conversation

@snmvaughan

@snmvaughan snmvaughan commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #7441.

Rationale for this change

With AQE on, a query fails with an AssertionError in AdaptiveSparkPlanExec.setLogicalLinkForNewQueryStage when the build side of a DPP broadcast contains a storage-partitioned join over native Iceberg scans.

Comet builds the DPP subquery's plan by wrapping the build side it already converted in a new AdaptiveSparkPlanExec, and that constructor runs EnsureRequirements over it again. Since #7231 the Comet joins, aggregates and windows declare Spark's required child distribution, and EnsureRequirements puts an exchange under each side of the join. AQE has no logical link for those exchanges.

The issue suggests having the native scan report the key-grouped partitioning of the scan it replaces, as #7424 did for SinglePartition. That is not enough for a join. In Spark 3.5 through 4.2, EnsureRequirements only treats two key-grouped children as co-partitioned when the parent is Spark's own SortMergeJoinExec or ShuffledHashJoinExec (checkKeyGroupCompatible), and in 4.1 every leaf below them must also be a KeyGroupedPartitionedScan (canPushDownSPJParamsToScan). A CometSortMergeJoinExec over CometIcebergNativeScanExecs is neither, so it finds no usable shuffle spec and shuffles both sides whatever the scans report. With only the scan change, the new join test fails in the same way.

An aggregate has one child, so EnsureRequirements only asks whether that child's partitioning satisfies the distribution. An aggregate above a storage-partitioned join does need the scans to report their partitioning. Without it the final aggregate gets an exchange between itself and the partial aggregate, inside a native block that was already serialized, and the native side fails with Column index 1 is out of bound.

What changes are included in this PR?

  • CometSortMergeJoinExec and CometHashJoinExec declare no required child distribution when both sides are storage-partitioned, meaning every leaf is a key-grouped scan and there is no shuffle above it. No shuffle sits below such a join for AQE to split, so the requirement from fix: declare required child distribution on aggregate, window and join operators #7231 protects nothing there. Every other join still returns the original operator's requirement. CometExec.isStoragePartitioned checks the join's current children, because conversion works bottom-up and originalPlan.children are already Comet nodes. It uses DataSourceV2ScanExecBase.keyGroupedPartitioning, which every supported Spark version declares, so it needs no shim even though Spark 4.2 renamed KeyGroupedPartitioning to KeyedPartitioning.
  • CometIcebergNativeScanExec.outputPartitioning reports the partitioning of the BatchScanExec it replaces, instead of always UnknownPartitioning(0), as CometNativeScanExec does for a bucketed scan. Reading it resolves no DPP subquery, because in Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 BatchScanExec.outputPartitioning is built from inputPartitions and only inputRDD uses filteredPartitions.
  • Four tests in CometIcebergNativeSuite, generated from two queries and two join hints. One query is the reproduction from the issue and the other puts an aggregate above the same storage-partitioned join. The MERGE hint gives a CometSortMergeJoinExec and SHUFFLE_HASH a CometHashJoinExec. Each case asserts that the plan has the expected join operator, that the fact scan carries a DPP filter, and that neither the plan nor the DPP subquery's plan has an exchange. The tables are dropped in a finally, so a failing case does not make the next one fail on CREATE TABLE.

How are these changes tested?

CometIcebergNativeSuite has four cases for this: the join and the aggregate, each with the MERGE and the SHUFFLE_HASH hint. They were restructured during review, so CI is the first run of the four as they are now.

Before that, on Spark 4.1:

  • The join test failed with only the scan change, and passed with the join change.
  • The aggregate test failed without the scan change, with Column index 1 is out of bound, and passed with it.
  • The 17 other AQE DPP tests in CometIcebergNativeSuite passed.

A reviewer ran the SHUFFLE_HASH join case on Spark 4.1 against main's operators.scala, where it fails with the same AssertionError, and it passes with this PR.

The change compiles on Spark 3.5 with Scala 2.12 and on Spark 4.1 with its tests, and its main sources compile on Spark 4.2. I have not run the tests on Spark 3.5 or 4.0.

AQE plans a DPP subquery's converted build side again, and EnsureRequirements
then puts an exchange under each side of a Comet join that Spark planned as a
storage-partitioned join, because the join declares the original operator's
required distribution since apache#7231. AQE has no logical link for those
exchanges and asserts.

Spark only accepts a key-grouped join as co-partitioned when the parent is its
own SortMergeJoinExec or ShuffledHashJoinExec, and in Spark 4.1 when every
leaf is also a KeyGroupedPartitionedScan. Reporting the scan's partitioning
therefore cannot help a join. Declare no required distribution on
CometSortMergeJoinExec and CometHashJoinExec when both sides are
storage-partitioned.

An aggregate above such a join does need the scan's partitioning, or the final
aggregate gets an exchange inside an already serialized native block. So
CometIcebergNativeScanExec now reports whatever the BatchScanExec it replaces
reports, which generalizes apache#7424 from SinglePartition.

Add a test for the join and one for an aggregate above it.

Closes apache#7441.

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @snmvaughan for tracking this down, and for the clear explanation of why the scan change alone can't fix the join. #7510 brought #7231 to branch-1.2, so branch-1.2 has this regression too, and I'll add the backport-1.2 label.

Comment on lines +782 to +783
// Spark's own scans. Comet's non-native V2 scan, CometBatchScanExec, is a leaf under a
// CometScanWrapper that this does not unwrap, so a join over it keeps its requirement.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment describes a path that no scan takes today. CometBatchScanExec is only created for CSV, Text, and Iceberg (CometScanRule.scala L415, L574, L1345), and CometExecRule handles each of those in its own case before the CometScanWrapper one (CometExecRule.scala L470-L483). An Iceberg scan that the native conversion declines stays a bare CometBatchScanExec. That class extends DataSourceV2ScanExecBase, so it matches the case below this comment and isn't excluded.

Could we reword the comment to match? The paragraph in the PR description about Comet's non-native Iceberg reader has the same problem, so it could go too.

Suggested change
// Spark's own scans. Comet's non-native V2 scan, CometBatchScanExec, is a leaf under a
// CometScanWrapper that this does not unwrap, so a join over it keeps its requirement.
// Spark's own scans, and CometBatchScanExec, which extends DataSourceV2ScanExecBase.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, thank you. CometBatchScanExec is only built for CSV, Text and Iceberg, and CometExecRule handles each of them before the CometScanWrapper case, so a declined Iceberg conversion stays a bare CometBatchScanExec that this case already matches. I used your wording for the comment and removed the paragraph from the description. Done in 7835efe.

val query =
"""SELECT /*+ BROADCAST(d) */ f.store_id, f.amount, d.name
|FROM aqe_cat.db.spj_dpp_fact f JOIN (
| SELECT /*+ MERGE(a, b) */ a.id, b.name

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CometHashJoinExec gets the new trait too, but both tests use MERGE. Could we also run them with SHUFFLE_HASH(a, b) and assert that the plan has a CometHashJoinExec? I tried that variant of the join test locally on Spark 4.1. It fails with the same AssertionError in setLogicalLinkForNewQueryStage when operators.scala is reverted to main, and passes with this PR.

The two tests also share all of their setup, including the table names. When I ran them without the operators.scala change, the aggregate test failed with TABLE_OR_VIEW_ALREADY_EXISTS on spj_dpp_fact, because the join test failed before its DROP TABLE lines ran. So a failure in one test hides the result of the other. Could we use one setup with the drops in a finally, and generate the cases with a for over the two queries and the two hints, like the test at L6170-L6174? That would also give the aggregate query the no-exchange assertion that only the join query has now.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 7835efe. The two queries and the two hints are now generated with a for, the tables are created once per case and dropped in a finally, and every case asserts the join operator the hint should produce (CometSortMergeJoinExec for MERGE, CometHashJoinExec for SHUFFLE_HASH), that the fact scan carries a DPP filter, and that neither the plan nor the DPP subquery's plan has an exchange, which the aggregate query now gets too. Thank you for running the SHUFFLE_HASH variant against a reverted operators.scala.

Generate the join and the aggregate test for the MERGE and SHUFFLE_HASH
hints, so CometSortMergeJoinExec and CometHashJoinExec are both covered.
Each case asserts the join operator, that the fact scan carries a DPP
filter, and that neither the plan nor the DPP subquery's plan has an
exchange. The tables are created once per case and dropped in a finally,
so a failing case no longer makes the next one fail on CREATE TABLE.

Correct the comment in isStoragePartitioned. A declined Iceberg scan
stays a bare CometBatchScanExec, which extends DataSourceV2ScanExecBase
and is matched by that case.

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked the EnsureRequirements reasoning against Spark 3.5.8, 4.0.1 and 4.1.1. checkKeyGroupCompatible only accepts Spark's SortMergeJoinExec or ShuffledHashJoinExec as the parent, so dropping the requirement on the Comet join is the change that can work. serializePartitions builds from inputRDD, which pads pruned key groups, so the reported partition count matches what runs.

Comment on lines +152 to +155
originalPlan.outputPartitioning match {
case _: UnknownPartitioning => UnknownPartitioning(0)
case reported => reported
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When BatchScanExec reports no partitioning, it returns UnknownPartitioning(0) from SparkPlan.outputPartitioning on every version I checked (v3.4.3, v3.5.8, v4.0.1, v4.1.1, v4.2.1). So this match always returns its input. CometNativeScanExec.outputPartitioning returns originalPlan.outputPartitioning directly for a bucketed scan, and this could do the same:

Suggested change
originalPlan.outputPartitioning match {
case _: UnknownPartitioning => UnknownPartitioning(0)
case reported => reported
}
originalPlan.outputPartitioning

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 6dab540, thank you. I checked DataSourceV2ScanExecBase.outputPartitioning on Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0, and its fallback is super.outputPartitioning in each, so the match was dead code. The originalPlan == null guard stays, because a canonicalized copy has no original plan.

…ed-dpp-join-exchanges

# Conflicts:
#	spark/src/main/scala/org/apache/spark/sql/comet/CometIcebergNativeScanExec.scala
BatchScanExec reports UnknownPartitioning(0) when it has no partitioning, on
every supported Spark version, so mapping an unknown partitioning to
UnknownPartitioning(0) changed nothing. Return the replaced scan's
outputPartitioning directly, as CometNativeScanExec does for a bucketed scan.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Iceberg area:scan Parquet scan / data reading backport-1.2 bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native Iceberg scan with AQE DPP fails when the DPP build side is a storage-partitioned join

3 participants