Skip to content
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.catalyst.rules.Rule
import org.apache.spark.sql.catalyst.util.sideBySide
import org.apache.spark.sql.comet.{CometCollectLimitExec, CometColumnarToRowExec, CometIcebergWriteExec, CometMapInBatchExec, CometNativeWriteExec, CometPlan, CometRangeExec, CometSparkToColumnarExec}
import org.apache.spark.sql.comet.execution.shuffle.{CometColumnarShuffle, CometShuffleExchangeExec}
import org.apache.spark.sql.comet.execution.shuffle.{CometColumnarShuffle, CometNativeShuffle, CometShuffleExchangeExec}
import org.apache.spark.sql.comet.shims.{MapInBatchInfo, ShimCometMapInBatch}
import org.apache.spark.sql.comet.util.Utils.containsVariantType
import org.apache.spark.sql.execution.{ColumnarToRowExec, RowToColumnarExec, SparkPlan}
Expand Down Expand Up @@ -176,6 +176,15 @@ case class EliminateRedundantTransitions(session: SparkSession)
CometColumnarShuffle,
_) =>
s.withNewChildren(Seq(child))

// Spark also adds `RowToColumnar` even when comet chose native shuffle.
// Native shuffle reads only Comet vectors, not Spark's vectors.
// Switch to row and drop the transition.
Comment on lines +180 to +182

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This adds a second way for a native exchange to end up as CometColumnarExchange, and a second producer of the Spark RowToColumnarExec that native shuffle cannot read. The input-contract paragraph in docs/source/contributor-guide/native_shuffle.md (just before "When Native Shuffle is Used") only describes the RevertNativeForTransitionHeavyStages case, and the "When JVM Shuffle is Used" list in jvm_shuffle.md does not have this one. Could you add it to both pages in this PR? Also, "Switch to the JVM columnar shuffle, which reads the child's rows" would be clearer here than "Switch to row".

case s @ CometShuffleExchangeExec(_, RowToColumnarExec(child), _, _, CometNativeShuffle, _)
if CometShuffleExchangeExec.columnarShuffleSupports(s, child) =>
Comment on lines +183 to +184

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When columnarShuffleSupports is false this arm does not match. The native exchange keeps RowToColumnarExec as its child, and every task fails, now with the new IllegalStateException. That case is not rare. columnarShuffleFailureReasons rejects spark.comet.shuffle.mode=native up front, and under the Celeborn manager a Comet exchange only exists in native mode, so the guard is false for every native-mode plan and every Celeborn plan. Could the declined case swap the transition for CometSparkToColumnarExec(child) instead, when CometSparkToColumnarExec.isSchemaSupported accepts the schema? That is how RevertNativeForTransitionHeavyStages.revertStageIfNeeded bridges a reverted stage under a native shuffle (outputArrow), and it keeps native shuffle. The JVM columnar shuffle can stay the first choice, since it avoids the string-key placement difference that hashesDifferentlyFromSpark describes. I have not checked whether the Arrow stream input works under Celeborn, so that case may need to stay as it is.

val columnar = s.copy(child = child, shuffleType = CometColumnarShuffle)
columnar.copyTagsFrom(s)
columnar
Comment on lines +185 to +187

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rewrite turns CometExchange into CometColumnarExchange without saying why. Could this tag the new node with withInfo(columnar, ...), saying the child became row-based after native shuffle was chosen? The verbose extended explain prints that as [COMET-INFO: ...]. Today the only plan that reaches this arm comes from EMR's nested AQE plan, so the explain output is where someone debugging it will look first.

}

eliminatedPlan match {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,12 @@ object CometArrowStream extends Logging {
val first = buffered.head
val expectedFields = expected.getFields
val actualFields = (0 until first.numCols()).map { i =>
val col = first.column(i).asInstanceOf[CometVector]
val col = first.column(i) match {
case c: CometVector => c
case other =>
throw new IllegalStateException(
s"$name expects CometVector type, but column $i is a ${other.getClass.getName}")
}

// The Arrow C Stream exports the producer's actual Arrow schema. If the
// consumer-declared schema has fewer fields, fall back to the producer's
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -575,6 +575,15 @@ object CometShuffleExchangeExec
nativeShuffleFailureReasons(s).isEmpty &&
!hashesDifferentlyFromSpark(s)

/**
* Whether the JVM columnar shuffle can take `s` with `rowChild`, a row-based plan, as its
* child. Pure: does not tag the node
*/
def columnarShuffleSupports(s: CometShuffleExchangeExec, rowChild: SparkPlan): Boolean =
!isCometCelebornShuffleManagerEnabled(s.conf) &&
columnarShuffleFailureReasons(
ShuffleExchangeExec(s.outputPartitioning, rowChild, s.shuffleOrigin)).isEmpty
Comment on lines +582 to +585

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can the isCometCelebornShuffleManagerEnabled check ever decide the result? Under Celeborn, shuffleSupported declines every exchange unless the mode is native (through cometCelebornShuffleFallbackReason), and for native mode columnarShuffleFailureReasons already returns "Comet columnar shuffle not enabled". If that holds, the helper is just columnarShuffleFailureReasons(...).isEmpty.


/**
* Whether the shuffle hashes into more than one partition a key that native shuffle, reading
* converted rows, may put in a different partition from Spark's partitioner. Native shuffle
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,13 @@

package org.apache.comet.rules

import org.apache.spark.SparkException
import org.apache.spark.sql.CometTestBase
import org.apache.spark.sql.catalyst.plans.physical.HashPartitioning
import org.apache.spark.sql.comet.{CometColumnarToRowExec, CometPlan}
import org.apache.spark.sql.execution.{ColumnarToRowExec, SparkPlan, UnionExec}
import org.apache.spark.sql.comet.execution.shuffle.{CometColumnarShuffle, CometNativeShuffle, CometShuffleExchangeExec}
import org.apache.spark.sql.execution.{ColumnarToRowExec, RowToColumnarExec, SparkPlan, UnionExec}
import org.apache.spark.sql.execution.exchange.ShuffleExchangeExec

import org.apache.comet.CometConf

Expand Down Expand Up @@ -64,4 +68,31 @@ class EliminateRedundantTransitionsSuite extends CometTestBase {
s"Spark-only branch should be left untouched:\n${result.treeString}")
}
}

// A native shuffle reads Arrow-backed Comet vectors. If its child turns row-based after Comet
// chose native shuffle, Spark add its own `RowToColumnar` under it, whose vectors are unable
// to be read.
test("native shuffle over Spark's RowToColumnar moves to the JVM columnar shuffle") {
val rowChild = sparkPlanFor("SELECT id, CAST(id AS STRING) AS s FROM range(10)")
assert(!rowChild.supportsColumnar)
val partitioning = HashPartitioning(rowChild.output.take(1), 4)
val shuffle = CometShuffleExchangeExec(
partitioning,
RowToColumnarExec(rowChild),
ShuffleExchangeExec(partitioning, rowChild),
shuffleType = CometNativeShuffle)

val result = EliminateRedundantTransitions(spark).apply(shuffle)

result match {
case CometShuffleExchangeExec(_, child, _, _, CometColumnarShuffle, _) =>
assert(child == rowChild, s"RowToColumnar should be removed:\n${result.treeString}")
case _ =>
fail(s"Expected the JVM columnar shuffle:\n${result.treeString}")
}
Comment on lines +85 to +92

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test only takes the branch where columnarShuffleSupports is true. Could you add a case under withSQLConf(CometConf.COMET_SHUFFLE_MODE.key -> "native") that pins what the rule does when the JVM columnar shuffle is not available? Today that is "plan unchanged", or with the bridge suggested above, a native exchange over CometSparkToColumnarExec.

assert(result.executeColumnar().map(_.numRows()).sum() == 10)

val e = intercept[SparkException](shuffle.executeColumnar().count())
assert(e.getMessage.contains("expects CometVector type"), e.getMessage)
}
}