Skip to content

fix: move native shuffle over Spark's RowToColumnar to the JVM columnar shuffle - #7704

Open
haziqishere wants to merge 10 commits into
apache:mainfrom
haziqishere:fix/native-shuffle-r2c
Open

haziqishere wants to merge 10 commits into
apache:mainfrom
haziqishere:fix/native-shuffle-r2c

Conversation

@haziqishere

Copy link
Copy Markdown

Which issue does this PR close?

Closes #7703

Rationale for this change

Native shuffle reads Arrow-backed CometVectors, but CometShuffleExchangeExec accepts any columnar child. When the child turns row-based after Comet chose native shuffle, Spark inserts RowToColumnarExec under the exchange. EliminateRedundantTransitions removes that transition under a JVM columnar shuffle, but has no case for native shuffle, so reconcileStreamSchema gets Spark's OnHeapColumnVector/OffHeapColumnVector and fails every task with a ClassCastException.

This happens on Amazon EMR, whose AQE wraps a reused exchange's child in a nested AdaptiveSparkPlanExec that reports row-based output. I haven't found an Apache Spark 4.1 plan that reaches it, so on Apache Spark this is hardening; the test builds the plan by hand and gets the same exception. #6607 handled the same cast for a different trigger (transitionRevert); #6685 is a different failure in the same code path, with Comet's own converter as the child.

What changes are included in this PR?

  • EliminateRedundantTransitions: a native CometShuffleExchangeExec over RowToColumnarExec drops the transition and switches to CometColumnarShuffle, which reads rows. Tags are copied to the new node, as RevertNativeForTransitionHeavyStages does for the same rewrite.
  • CometShuffleExchangeExec.columnarShuffleSupports: new helper that guards the rewrite with the existing JVM columnar shuffle checks (Celeborn, columnarShuffleFailureReasons). When it is false, the plan is left as is.
  • CometArrowStream.reconcileStreamSchema: a column that is not a CometVector now throws an IllegalStateException naming its class, instead of a bare ClassCastException. No config or public API changes.

How are these changes tested?

New test in EliminateRedundantTransitionsSuite builds CometShuffleExchangeExec(CometNativeShuffle) over RowToColumnarExec(rowChild) and checks that:

  • the rule returns a JVM columnar shuffle over rowChild,
  • the rewritten plan executes and returns all rows,
  • executing the original plan fails with the new error.

Without the rule change, the test fails on the first check. Without the reconcileStreamSchema change, executing the original plan on Apache Spark 4.1 fails with the exception seen on EMR:

java.lang.ClassCastException: class org.apache.spark.sql.execution.vectorized.OffHeapColumnVector cannot be cast to class org.apache.comet.vector.CometVector

Ran locally (Spark 4.1 profile, JDK 17): EliminateRedundantTransitionsSuite, RevertNativeForTransitionHeavyStagesSuite, CometNativeShuffleSuite.

Local test output
- native shuffle over Spark's RowToColumnar moves to the JVM columnar shuffle (594 milliseconds)
Tests: succeeded 128, failed 0, canceled 0, ignored 0, pending 0
BUILD SUCCESS

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tracking this down and for the hand-built repro. Moving the exchange to the JVM columnar shuffle is a good first choice, because Spark's own partitioner then places the rows. When that shuffle is not available (spark.comet.shuffle.mode=native, which also covers every Celeborn plan) the rule still leaves a plan that fails every task, and the shuffle contributor docs do not mention the new case yet. No CI has run on this head (CI (fork) found no run), and since this is a planner and shuffle change it needs a Spark SQL suite verdict, from dev/local-ci.sh spark or a run-spark-* label, before it is queued.

Comment on lines +183 to +184
case s @ CometShuffleExchangeExec(_, RowToColumnarExec(child), _, _, CometNativeShuffle, _)
if CometShuffleExchangeExec.columnarShuffleSupports(s, child) =>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When columnarShuffleSupports is false this arm does not match. The native exchange keeps RowToColumnarExec as its child, and every task fails, now with the new IllegalStateException. That case is not rare. columnarShuffleFailureReasons rejects spark.comet.shuffle.mode=native up front, and under the Celeborn manager a Comet exchange only exists in native mode, so the guard is false for every native-mode plan and every Celeborn plan. Could the declined case swap the transition for CometSparkToColumnarExec(child) instead, when CometSparkToColumnarExec.isSchemaSupported accepts the schema? That is how RevertNativeForTransitionHeavyStages.revertStageIfNeeded bridges a reverted stage under a native shuffle (outputArrow), and it keeps native shuffle. The JVM columnar shuffle can stay the first choice, since it avoids the string-key placement difference that hashesDifferentlyFromSpark describes. I have not checked whether the Arrow stream input works under Celeborn, so that case may need to stay as it is.

Comment on lines +185 to +187
val columnar = s.copy(child = child, shuffleType = CometColumnarShuffle)
columnar.copyTagsFrom(s)
columnar

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rewrite turns CometExchange into CometColumnarExchange without saying why. Could this tag the new node with withInfo(columnar, ...), saying the child became row-based after native shuffle was chosen? The verbose extended explain prints that as [COMET-INFO: ...]. Today the only plan that reaches this arm comes from EMR's nested AQE plan, so the explain output is where someone debugging it will look first.

Comment on lines +180 to +182
// Spark also adds `RowToColumnar` even when comet chose native shuffle.
// Native shuffle reads only Comet vectors, not Spark's vectors.
// Switch to row and drop the transition.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This adds a second way for a native exchange to end up as CometColumnarExchange, and a second producer of the Spark RowToColumnarExec that native shuffle cannot read. The input-contract paragraph in docs/source/contributor-guide/native_shuffle.md (just before "When Native Shuffle is Used") only describes the RevertNativeForTransitionHeavyStages case, and the "When JVM Shuffle is Used" list in jvm_shuffle.md does not have this one. Could you add it to both pages in this PR? Also, "Switch to the JVM columnar shuffle, which reads the child's rows" would be clearer here than "Switch to row".

Comment on lines +582 to +585
def columnarShuffleSupports(s: CometShuffleExchangeExec, rowChild: SparkPlan): Boolean =
!isCometCelebornShuffleManagerEnabled(s.conf) &&
columnarShuffleFailureReasons(
ShuffleExchangeExec(s.outputPartitioning, rowChild, s.shuffleOrigin)).isEmpty

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can the isCometCelebornShuffleManagerEnabled check ever decide the result? Under Celeborn, shuffleSupported declines every exchange unless the mode is native (through cometCelebornShuffleFallbackReason), and for native mode columnarShuffleFailureReasons already returns "Comet columnar shuffle not enabled". If that holds, the helper is just columnarShuffleFailureReasons(...).isEmpty.

Comment on lines +85 to +92
val result = EliminateRedundantTransitions(spark).apply(shuffle)

result match {
case CometShuffleExchangeExec(_, child, _, _, CometColumnarShuffle, _) =>
assert(child == rowChild, s"RowToColumnar should be removed:\n${result.treeString}")
case _ =>
fail(s"Expected the JVM columnar shuffle:\n${result.treeString}")
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test only takes the branch where columnarShuffleSupports is true. Could you add a case under withSQLConf(CometConf.COMET_SHUFFLE_MODE.key -> "native") that pins what the rule does when the JVM columnar shuffle is not available? Today that is "plan unchanged", or with the bridge suggested above, a native exchange over CometSparkToColumnarExec.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native shuffle fails with ClassCastException when its child is Spark's RowToColumnarExec

2 participants