Repository navigation
fix: move native shuffle over Spark's RowToColumnar to the JVM columnar shuffle - #7704
haziqishere wants to merge 10 commits into
Conversation
…ceive unexpected type
comphead
left a comment
There was a problem hiding this comment.
Thanks for tracking this down and for the hand-built repro. Moving the exchange to the JVM columnar shuffle is a good first choice, because Spark's own partitioner then places the rows. When that shuffle is not available (spark.comet.shuffle.mode=native, which also covers every Celeborn plan) the rule still leaves a plan that fails every task, and the shuffle contributor docs do not mention the new case yet. No CI has run on this head (CI (fork) found no run), and since this is a planner and shuffle change it needs a Spark SQL suite verdict, from dev/local-ci.sh spark or a run-spark-* label, before it is queued.
| case s @ CometShuffleExchangeExec(_, RowToColumnarExec(child), _, _, CometNativeShuffle, _) | ||
| if CometShuffleExchangeExec.columnarShuffleSupports(s, child) => |
There was a problem hiding this comment.
When columnarShuffleSupports is false this arm does not match. The native exchange keeps RowToColumnarExec as its child, and every task fails, now with the new IllegalStateException. That case is not rare. columnarShuffleFailureReasons rejects spark.comet.shuffle.mode=native up front, and under the Celeborn manager a Comet exchange only exists in native mode, so the guard is false for every native-mode plan and every Celeborn plan. Could the declined case swap the transition for CometSparkToColumnarExec(child) instead, when CometSparkToColumnarExec.isSchemaSupported accepts the schema? That is how RevertNativeForTransitionHeavyStages.revertStageIfNeeded bridges a reverted stage under a native shuffle (outputArrow), and it keeps native shuffle. The JVM columnar shuffle can stay the first choice, since it avoids the string-key placement difference that hashesDifferentlyFromSpark describes. I have not checked whether the Arrow stream input works under Celeborn, so that case may need to stay as it is.
| val columnar = s.copy(child = child, shuffleType = CometColumnarShuffle) | ||
| columnar.copyTagsFrom(s) | ||
| columnar |
There was a problem hiding this comment.
The rewrite turns CometExchange into CometColumnarExchange without saying why. Could this tag the new node with withInfo(columnar, ...), saying the child became row-based after native shuffle was chosen? The verbose extended explain prints that as [COMET-INFO: ...]. Today the only plan that reaches this arm comes from EMR's nested AQE plan, so the explain output is where someone debugging it will look first.
| // Spark also adds `RowToColumnar` even when comet chose native shuffle. | ||
| // Native shuffle reads only Comet vectors, not Spark's vectors. | ||
| // Switch to row and drop the transition. |
There was a problem hiding this comment.
This adds a second way for a native exchange to end up as CometColumnarExchange, and a second producer of the Spark RowToColumnarExec that native shuffle cannot read. The input-contract paragraph in docs/source/contributor-guide/native_shuffle.md (just before "When Native Shuffle is Used") only describes the RevertNativeForTransitionHeavyStages case, and the "When JVM Shuffle is Used" list in jvm_shuffle.md does not have this one. Could you add it to both pages in this PR? Also, "Switch to the JVM columnar shuffle, which reads the child's rows" would be clearer here than "Switch to row".
| def columnarShuffleSupports(s: CometShuffleExchangeExec, rowChild: SparkPlan): Boolean = | ||
| !isCometCelebornShuffleManagerEnabled(s.conf) && | ||
| columnarShuffleFailureReasons( | ||
| ShuffleExchangeExec(s.outputPartitioning, rowChild, s.shuffleOrigin)).isEmpty |
There was a problem hiding this comment.
Can the isCometCelebornShuffleManagerEnabled check ever decide the result? Under Celeborn, shuffleSupported declines every exchange unless the mode is native (through cometCelebornShuffleFallbackReason), and for native mode columnarShuffleFailureReasons already returns "Comet columnar shuffle not enabled". If that holds, the helper is just columnarShuffleFailureReasons(...).isEmpty.
| val result = EliminateRedundantTransitions(spark).apply(shuffle) | ||
|
|
||
| result match { | ||
| case CometShuffleExchangeExec(_, child, _, _, CometColumnarShuffle, _) => | ||
| assert(child == rowChild, s"RowToColumnar should be removed:\n${result.treeString}") | ||
| case _ => | ||
| fail(s"Expected the JVM columnar shuffle:\n${result.treeString}") | ||
| } |
There was a problem hiding this comment.
This test only takes the branch where columnarShuffleSupports is true. Could you add a case under withSQLConf(CometConf.COMET_SHUFFLE_MODE.key -> "native") that pins what the rule does when the JVM columnar shuffle is not available? Today that is "plan unchanged", or with the bridge suggested above, a native exchange over CometSparkToColumnarExec.
Which issue does this PR close?
Closes #7703
Rationale for this change
Native shuffle reads Arrow-backed
CometVectors, butCometShuffleExchangeExecaccepts any columnar child. When the child turns row-based after Comet chose native shuffle, Spark insertsRowToColumnarExecunder the exchange.EliminateRedundantTransitionsremoves that transition under a JVM columnar shuffle, but has no case for native shuffle, soreconcileStreamSchemagets Spark'sOnHeapColumnVector/OffHeapColumnVectorand fails every task with aClassCastException.This happens on Amazon EMR, whose AQE wraps a reused exchange's child in a nested
AdaptiveSparkPlanExecthat reports row-based output. I haven't found an Apache Spark 4.1 plan that reaches it, so on Apache Spark this is hardening; the test builds the plan by hand and gets the same exception. #6607 handled the same cast for a different trigger (transitionRevert); #6685 is a different failure in the same code path, with Comet's own converter as the child.What changes are included in this PR?
EliminateRedundantTransitions: a nativeCometShuffleExchangeExecoverRowToColumnarExecdrops the transition and switches toCometColumnarShuffle, which reads rows. Tags are copied to the new node, asRevertNativeForTransitionHeavyStagesdoes for the same rewrite.CometShuffleExchangeExec.columnarShuffleSupports: new helper that guards the rewrite with the existing JVM columnar shuffle checks (Celeborn,columnarShuffleFailureReasons). When it is false, the plan is left as is.CometArrowStream.reconcileStreamSchema: a column that is not aCometVectornow throws anIllegalStateExceptionnaming its class, instead of a bareClassCastException. No config or public API changes.How are these changes tested?
New test in
EliminateRedundantTransitionsSuitebuildsCometShuffleExchangeExec(CometNativeShuffle)overRowToColumnarExec(rowChild)and checks that:rowChild,Without the rule change, the test fails on the first check. Without the
reconcileStreamSchemachange, executing the original plan on Apache Spark 4.1 fails with the exception seen on EMR:Ran locally (Spark 4.1 profile, JDK 17):
EliminateRedundantTransitionsSuite,RevertNativeForTransitionHeavyStagesSuite,CometNativeShuffleSuite.Local test output