Skip to content

Native shuffle fails with ClassCastException when its child is Spark's RowToColumnarExec #7703

Open

Description

@haziqishere

Describe the bug

TL;DR: I got ClassCastException during shuffle stage. This is because CometNativeShuffle expects type CometVector but receive Spark's OffHeapColumnVector instead. This is caused by Spark inserted RowToColumnar. This behaviour is seen on Amazon EMR, where AQE wraps a reused exchange's child in a nested AdaptivesparkPlanExec that reports row-based output, hence Spark adds RowToColumnar.

My Observation

A CometShuffleExchangeExec with shuffleType = CometNativeShuffle whose child is Spark's RowToColumnarExec fails every task, so the stage aborts:

  java.lang.ClassCastException: class org.apache.spark.sql.execution.vectorized.OffHeapColumnVector
    cannot be cast to class org.apache.comet.vector.CometVector
    at org.apache.spark.sql.comet.execution.arrow.CometArrowStream$.reconcileStreamSchema(CometNativeArrowSource.scala:164)
    at org.apache.spark.sql.comet.execution.arrow.CometArrowStream$.wrapColumnarBatchRDD(CometNativeArrowSource.scala:107)
    at org.apache.spark.sql.comet.CometExecRDD$.resolveInputObjects(CometExecRDD.scala:207)
    at org.apache.spark.sql.comet.execution.shuffle.CometNativeShuffleInputRDD.compute(CometNativeShuffleInputRDD.scala:120)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(...)

Native shuffle reads Arrow-backed CometVectors. When the exchange's child turns row-based after Comet chose native shuffle, Spark's ApplyColumnarRulesAndInsertTransitions puts a RowToColumnarExec under it. EliminateRedundantTransitions removes that transition under a JVM columnar shuffle, but has no case for native shuffle, so it stays. At runtimeCometShuffleExchangeExec falls back to child.executeColumnar(), which returns Spark's OnHeapColumnVector/OffHeapColumnVector batches, and reconcileStreamSchema casts them to CometVector without checking.

The plan at failure:

  CometExchange hashpartitioning(acct, 200), ..., CometNativeShuffle
  +- RowToColumnar
     +- AdaptiveSparkPlan isFinalPlan=true
        +- ResultQueryStage
           +- CometColumnarToRow
              +- CometHashAggregate [Partial] partial_count(distinct other)

The same cast issue came up in the review of #6607, reached through transitionRevert with maxTransitions=0. That PR only fixed that path.

Steps to reproduce

On Amazon EMR (Spark 4.1, Comet 1.2.0), with AQE on: a subtree that contains a countDistinct and is used twice in the same query, so Spark reuses its exchange. A minimal shape:

from pyspark.sql import functions as F

spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "-1")
spark.conf.set("spark.sql.adaptive.autoBroadcastJoinThreshold", "-1")

spark.range(100000) \
    .select((F.col("id") % 1000).alias("k"), (F.col("id") % 37).alias("v")) \
    .write.mode("overwrite").parquet("/tmp/comet_r2c")
df = spark.read.parquet("/tmp/comet_r2c")

agg = df.groupBy("k").agg(F.countDistinct("v").alias("n"))  # two shuffles deep
agg.alias("a").join(agg.alias("b"), "k").count()            # same subtree twice -> reused exchange

EMR's AQE wraps the reused exchange's child in a nested AdaptiveSparkPlanExec. That nested plan reports row-based output by default (spark.sql.adaptive.nested.propagateColumnarSupport=false), so Spark inserts RowToColumnar under the native exchange. The query passes with AQE off, without the reuse (hubs used once), or with count in place of countDistinct.

On Apache Spark I haven't found a plan that reaches it: the same query builds the native exchange and the reused exchange, but never the nested AQE plan. Building the plan by hand reproduces the same exception on Apache Spark 4.1:

val rowChild = spark.sql("SELECT id, CAST(id AS STRING) AS s FROM range(10)")
  .queryExecution.executedPlan  // any row-based plan, with Comet disabled
val partitioning = HashPartitioning(rowChild.output.take(1), 4)
CometShuffleExchangeExec(
  partitioning,
  RowToColumnarExec(rowChild),
  ShuffleExchangeExec(partitioning, rowChild),
  shuffleType = CometNativeShuffle).executeColumnar().count()

Expected behavior

The exchange runs. The JVM columnar shuffle reads rows, so Comet can drop the RowToColumnar and use it instead, the same way #6607 handles its trigger. When that isn't possible, the error should name the vector class and the cause instead of a bare ClassCastException.

Additional context

  • My setup: Comet 1.2.0 on EMR (Spark 4.1). The code path is unchanged on main.
  • Workarounds: spark.comet.shuffle.mode=jvm, or on EMR spark.sql.adaptive.nested.propagateColumnarSupport=true, which keeps native shuffle.

Willingness to contribute

I would be willing to contribute a fix for this bug with guidance from the DataFusion community

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:shuffleShuffle (JVM and native)bugSomething isn't workingpriority:highCrashes, panics, segfaults, major functional breakage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions