Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
fd3ed02
feat: route MapSort fallback through the codegen dispatcher
Sep 9, 2026
0fa0f9c
test: add a matched MapSort dispatcher vs Spark-fallback microbenchmark
Sep 10, 2026
f1a7c2e
test: time the prepared MapSort physical plan instead of a noop write
Sep 11, 2026
0f0bcc1
test: strengthen MapSort dispatcher correctness coverage
Sep 14, 2026
5275b43
docs: clarify MapSort JVM routing and rollback
Sep 14, 2026
a74af9a
Merge branch 'apache:main' into feat-map-sort-codegen-dispatch-5590
sam-1112 Sep 17, 2026
d411c88
Merge branch 'main' into feat-map-sort-codegen-dispatch-5590
sam-1112 Sep 19, 2026
db2a64d
feat: make the MapSort codegen dispatcher opt-in by default
sam-1112 Sep 21, 2026
b5af75d
Merge branch 'main' into feat-map-sort-codegen-dispatch-5590
sam-1112 Sep 21, 2026
613efaf
Merge branch 'main' into feat-map-sort-codegen-dispatch-5590
sam-1112 Sep 22, 2026
04ba768
test: capture the target Parquet insert plan, not a delayed source IN…
sam-1112 Sep 22, 2026
2c868d8
test: qualify Parquet output paths before matching write plans
sam-1112 Sep 22, 2026
e29715f
Merge branch 'main' into feat-map-sort-codegen-dispatch-5590
sam-1112 Sep 22, 2026
b1fd92b
Merge remote-tracking branch 'upstream/main' into feat-map-sort-codeg…
sam-1112 Nov 15, 2026
23cb626
Merge remote-tracking branch 'origin/feat-map-sort-codegen-dispatch-5…
sam-1112 Nov 15, 2026
0a13f55
fix: align MapSort coverage with JVM shuffle routing
sam-1112 Nov 15, 2026
18968b6
fix: explicitly widen MapSort benchmark row counts to Long
sam-1112 Nov 16, 2026
3f03f5d
Merge remote-tracking branch 'upstream' into feat-map-sort-codegen-di…
sam-1112 Nov 18, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -21,14 +21,30 @@ under the License.

## MapSort (Spark 4.0+)

Spark 4.0 inserts `MapSort` to normalize map values when they appear in shuffle hash partitioning
keys, in `try_element_at`, and in other contexts where map ordering must be deterministic. Comet
runs `MapSort` natively, so map shuffle and group-by-on-map stay on Comet under Spark 4.0.

When `spark.comet.exec.strictFloatingPoint=true`, `MapSort` falls back to Spark for maps whose
keys contain `Float` or `Double`. Arrow's sort uses
IEEE total ordering for floating-point, which differs from Spark's `Double.compare` semantics for
`NaN` and `-0.0`.
Spark 4.0 inserts `MapSort` to normalize map values when they appear in grouping expressions or
shuffle hash partitioning keys. Comet runs `MapSort` in Rust for supported scalar key types.

Other orderable key types have no native implementation. By default those cases fall the
enclosing projection or native shuffle back to Spark. Set
`spark.comet.expression.MapSort.codegen.enabled=true` to run Spark's own generated JVM code through
the codegen dispatcher instead, so the enclosing operator can stay in the Comet pipeline. That
dispatcher route is not a native `MapSort` implementation. A matched microbenchmark of projections
and forced native shuffle is slower than Spark fallback for most measured shapes; it does not
compare JVM columnar shuffle with native shuffle in `auto` mode.

When `spark.comet.exec.strictFloatingPoint=true`, maps whose keys contain `Float` or `Double`
follow the same default for projections and native shuffle. The codegen setting above routes
those keys through the dispatcher instead.
Arrow's sort uses IEEE total ordering for floating-point, which differs from Spark's
`Double.compare` semantics for `NaN` and `-0.0`. If the dispatcher is disabled or cannot handle
an expression, Comet safely falls back to Spark.

Set `spark.comet.expression.MapSort.enabled=false` to disable Comet `MapSort` entirely, including
the Rust implementation for scalar keys. That expression-specific setting leaves the codegen
dispatcher available to unrelated expressions.

JVM shuffle computes partition IDs with Spark expressions, so it supports these map keys
independently of the `MapSort` codegen setting.

<!--BEGIN:EXPR_COMPAT[map]-->
<!--END:EXPR_COMPAT-->
18 changes: 18 additions & 0 deletions spark/src/main/scala/org/apache/comet/CometConf.scala
Original file line number Diff line number Diff line change
Expand Up @@ -856,6 +856,24 @@ object CometConf extends ShimCometConf {
.booleanConf
.createWithDefault(true)

val COMET_EXPRESSION_MAPSORT_CODEGEN_ENABLED: ConfigEntry[Boolean] =
conf("spark.comet.expression.MapSort.codegen.enabled")
.category(CATEGORY_EXEC)
.doc(
"Whether to route `MapSort` cases that have no native implementation through Comet's " +
"JVM codegen dispatcher (Spark's own `doGenCode` inside the Comet pipeline). This " +
"includes array, struct, interval, and non-default-collated string map keys, and " +
"floating-point keys under `spark.comet.exec.strictFloatingPoint=true`. Enabling it " +
"can keep projections and eligible native shuffles in Comet. JVM columnar shuffle " +
"computes partition IDs with Spark expressions independently of this setting. " +
"Disabled by default because a matched benchmark of projections and forced native " +
"shuffle is slower than Spark fallback for most measured shapes. Those results do " +
"not compare JVM columnar shuffle with native shuffle in `auto` mode. Supported " +
"scalar keys continue to use Rust `map_sort`. This flag does not change codegen " +
"dispatch for any other expression.")
.booleanConf
.createWithDefault(false)

val COMET_SHUFFLE_NATIVE_HASH_PARTITIONING_ENABLED: ConfigEntry[Boolean] =
conf("spark.comet.shuffle.native.partitioning.hash.enabled")
.withAlternative("spark.comet.native.shuffle.partitioning.hash.enabled")
Expand Down
49 changes: 39 additions & 10 deletions spark/src/main/scala/org/apache/comet/GenerateDocs.scala
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,12 @@ object GenerateDocs {
* retain the native opt-in behavior
* @param codegenDispatchFallback
* whether the serde mixes in `CodegenDispatchFallback`, meaning `unsupportedReasons` cases
* route through the JVM codegen dispatcher instead of falling back to Spark
* can route through the JVM codegen dispatcher instead of falling back to Spark
* @param codegenDispatchEnabledByDefault
* whether those dispatcher cases run by default. When false, they fall back to Spark until
* the user sets `codegenDispatchConfigKey`
* @param codegenDispatchConfigKey
* the config key that enables codegen dispatch when it is not the default
*/
private[comet] case class ExprNotes(
name: String,
Expand All @@ -71,7 +76,9 @@ object GenerateDocs {
nativeOptIn: Boolean,
nativeOptInConfigKey: String,
conditionalNativeDefault: Boolean,
codegenDispatchFallback: Boolean)
codegenDispatchFallback: Boolean,
codegenDispatchEnabledByDefault: Boolean = true,
codegenDispatchConfigKey: Option[String] = None)

private type CategoryNotes = Seq[ExprNotes]

Expand All @@ -94,7 +101,15 @@ object GenerateDocs {
case n: NativeOptInAvailable => n.hasConditionalNativeDefault
case _ => false
},
codegenDispatchFallback = serde.isInstanceOf[CodegenDispatchFallback])
codegenDispatchFallback = serde.isInstanceOf[CodegenDispatchFallback],
codegenDispatchEnabledByDefault = serde match {
case h: CodegenDispatchFallback => h.codegenDispatchEnabledByDefault
case _ => true
},
codegenDispatchConfigKey = serde match {
case h: CodegenDispatchFallback => h.codegenDispatchConfigKey
case _ => None
})
}

/** Build the documentation notes for a single aggregate expression serde. */
Expand Down Expand Up @@ -422,6 +437,13 @@ object GenerateDocs {
" (run inside the Comet pipeline) by default." +
s" Set `${n.nativeOptInConfigKey}=true` to explicitly select Comet's native" +
" implementation, which has the following differences from Spark:\n\n"
} else if (n.codegenDispatchFallback && !n.codegenDispatchEnabledByDefault) {
val dispatchKey =
n.codegenDispatchConfigKey.getOrElse(s"spark.comet.expression.$name.codegen.enabled")
s"\nThe following incompatibilities cause `$name` to fall back to Spark by default." +
s" Set `$dispatchKey=true` to run Spark's code-generated implementation inside the" +
s" Comet pipeline. Set `${n.nativeOptInConfigKey}=true` to enable Comet's native" +
" implementation despite these differences.\n\n"
} else if (n.nativeOptIn) {
s"\nBy default, `$name` is evaluated in the JVM using Spark's own code-generated" +
" implementation (run inside the Comet pipeline), which matches Spark exactly." +
Expand All @@ -438,13 +460,20 @@ object GenerateDocs {
}
}
if (n.unsupportedReasons.nonEmpty) {
val header = if (n.codegenDispatchFallback) {
"\nThe following cases have no native implementation and always run in the JVM using" +
" Spark's code-generated implementation (inside the Comet pipeline):\n\n"
} else {
"\nThe following cases are not supported by Comet and always fall back to Spark," +
" regardless of any `allowIncompatible` setting:\n\n"
}
val header =
if (n.codegenDispatchFallback && n.codegenDispatchEnabledByDefault) {
"\nThe following cases have no native implementation and always run in the JVM using" +
" Spark's code-generated implementation (inside the Comet pipeline):\n\n"
} else if (n.codegenDispatchFallback) {
val dispatchKey = n.codegenDispatchConfigKey.getOrElse(
s"spark.comet.expression.$name.codegen.enabled")
"\nThe following cases have no native implementation and fall back to Spark by" +
s" default. Set `$dispatchKey=true` to run Spark's code-generated implementation" +
" inside the Comet pipeline:\n\n"
} else {
"\nThe following cases are not supported by Comet and always fall back to Spark," +
" regardless of any `allowIncompatible` setting:\n\n"
}
output.append(header)
for (reason <- n.unsupportedReasons) {
output.append(s"- $reason\n")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,10 @@ trait CometExpressionSerde[T <: Expression] {
* for the case where the dispatcher itself cannot handle the expression (e.g. the global codegen
* flag is off, or the kernel rejects the bound tree).
*
* A serde may override [[isCodegenDispatchEnabled]] so that this mixin is a no-op until the user
* opts in. When that method returns `false`, `Unsupported` and (non-opt-in) `Incompatible` fall
* back to Spark the same way a serde without this mixin would.
*
* Contract for `Unsupported` reasons on a `CodegenDispatchFallback` serde: the case must be
* something `Expression.doGenCode` can compile. If you mark something `Unsupported` because Spark
* also rejects it, that is fine -- the dispatcher will surface the same error Spark would have.
Expand All @@ -122,7 +126,27 @@ trait CometExpressionSerde[T <: Expression] {
* dispatcher. Every other `Incompatible` expression falls back to Spark, and every other
* `Unsupported` expression falls back to Spark.
*/
trait CodegenDispatchFallback extends NativeOptInAvailable { self: CometExpressionSerde[_] => }
trait CodegenDispatchFallback extends NativeOptInAvailable { self: CometExpressionSerde[_] =>

/**
* Whether `Unsupported` / non-opt-in `Incompatible` cases should run through the JVM codegen
* dispatcher. Defaults to true. Override to gate the dispatcher behind a config; when false,
* those cases fall back to Spark.
*/
def isCodegenDispatchEnabled: Boolean = true

/**
* Whether [[isCodegenDispatchEnabled]] is true at the config default. Documentation metadata
* only; does not affect runtime routing.
*/
def codegenDispatchEnabledByDefault: Boolean = true

/**
* Config key the user sets to enable codegen dispatch when [[codegenDispatchEnabledByDefault]]
* is false. Documentation metadata only.
*/
def codegenDispatchConfigKey: Option[String] = None
}

/**
* Marker for serdes that have a native implementation the user can opt into. Normally these
Expand Down
13 changes: 7 additions & 6 deletions spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala
Original file line number Diff line number Diff line change
Expand Up @@ -1417,18 +1417,19 @@ object QueryPlanSerde extends Logging with CometExprShim with CometTypeShim {
}

/**
* If `handler` is a `CodegenDispatchFallback`, run `expr` through the JVM codegen dispatcher
* and return `Some((handler, proto))` on success; otherwise return `None`. Shared by the
* `Unsupported` and (non-opt-in) `Incompatible` arms of `exprToProtoInternal` so they don't
* each inline the same pattern match. Returning the matched handler lets the `Incompatible` arm
* reach `nativeOptInConfigKeyOverride` without re-pattern-matching the same value.
* If `handler` is a `CodegenDispatchFallback` whose dispatcher is enabled, run `expr` through
* the JVM codegen dispatcher and return `Some((handler, proto))` on success; otherwise return
* `None`. Shared by the `Unsupported` and (non-opt-in) `Incompatible` arms of
* `exprToProtoInternal` so they don't each inline the same pattern match. Returning the matched
* handler lets the `Incompatible` arm reach `nativeOptInConfigKeyOverride` without
* re-pattern-matching the same value.
*/
private def dispatchIfFallback(
handler: CometExpressionSerde[_],
expr: Expression,
inputs: Seq[Attribute],
binding: Boolean): Option[(CodegenDispatchFallback, Expr)] = handler match {
case h: CodegenDispatchFallback =>
case h: CodegenDispatchFallback if h.isCodegenDispatchEnabled =>
CometScalaUDF.emitJvmCodegenDispatch(expr, inputs, binding).map(h -> _)
case _ => None
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -638,9 +638,13 @@ object CometShuffleExchangeExec
case MapType(keyType, valueType, _) if nestedHashPartitioningEnabled =>
// Map entry order is not semantically meaningful, so two equal maps must hash alike.
// Spark 4.0+ normalizes a map shuffle key by wrapping it in `mapsort(...)`, which is
// gated separately by CometMapSort (scalar map keys only) and, when unsupported, fails
// the expression check below. Earlier Spark versions insert no such normalization, so
// Comet would hash physical entry order and could route equal maps differently.
// gated separately by CometMapSort. Scalar keys use native map_sort. Other orderable key
// types rule out native shuffle unless
// `spark.comet.expression.MapSort.codegen.enabled` is set, in which case Spark's generated
// MapSort code runs through the JVM dispatcher. JVM columnar shuffle remains eligible
// independently of this gate. Earlier Spark versions insert no such
// normalization, so Comet would hash physical entry order and could route equal maps
// differently.
isSpark40Plus &&
supportedHashPartitioningDataType(keyType) &&
supportedHashPartitioningDataType(valueType)
Expand Down
24 changes: 21 additions & 3 deletions spark/src/main/spark-4.x/org/apache/comet/serde/CometMapSort.scala
Original file line number Diff line number Diff line change
Expand Up @@ -25,20 +25,38 @@ import org.apache.spark.sql.types.MapType
import org.apache.comet.CometConf
import org.apache.comet.serde.QueryPlanSerde.{exprToProtoInternal, scalarFunctionExprToProtoWithReturnType, supportedScalarSortElementType}

object CometMapSort extends CometExpressionSerde[MapSort] {
// Key types without a native implementation can run Spark's generated code in-pipeline when
// spark.comet.expression.MapSort.codegen.enabled=true. That dispatcher is off by default: a
// matched microbenchmark of those shapes is slower than falling the enclosing operator back to
// Spark. Spark rejects collated-string map keys by default, but they reach MapSort when
// spark.sql.collation.allowInMapKeys=true and the map is built from dispatcher-supported inputs;
// with the codegen setting on, those expressions take the same Unsupported -> dispatcher route.
// A scan carrying a collated map schema may still be rejected independently by the scan's schema
// support checks.
object CometMapSort extends CometExpressionSerde[MapSort] with CodegenDispatchFallback {

override def isCodegenDispatchEnabled: Boolean =
CometConf.COMET_EXPRESSION_MAPSORT_CODEGEN_ENABLED.get()

override def codegenDispatchEnabledByDefault: Boolean = false

override def codegenDispatchConfigKey: Option[String] =
Some(CometConf.COMET_EXPRESSION_MAPSORT_CODEGEN_ENABLED.key)

override def getIncompatibleReasons(): Seq[String] =
Seq(
"MapSort on floating-point keys is not 100% compatible with Spark when " +
s"`${CometConf.COMET_EXEC_STRICT_FLOATING_POINT.key}=true`.")

override def getUnsupportedReasons(): Seq[String] =
Seq("MapSort is unsupported for non-scalar key types (struct, array, map, etc.).")
Seq(
"MapSort with an orderable key type outside native scalar coverage, including array, " +
"struct, interval, and non-default-collated string keys, has no native implementation.")

override def getSupportLevel(expr: MapSort): SupportLevel = {
val keyType = expr.dataType.asInstanceOf[MapType].keyType
if (!supportedScalarSortElementType(keyType)) {
Unsupported(Some(s"MapSort on map with key type $keyType is not supported"))
Unsupported(Some(s"MapSort with key type $keyType has no native implementation"))
} else {
SupportLevel
.strictFloatingPointReason(keyType, "MapSort on floating-point key")
Expand Down
14 changes: 12 additions & 2 deletions spark/src/test/scala/org/apache/comet/CometCodegenAssertions.scala
Original file line number Diff line number Diff line change
Expand Up @@ -33,13 +33,23 @@ import org.apache.comet.vector.CometVector
trait CometCodegenAssertions {

/** Asserts the dispatcher actually ran during `f`, guarding against silent serde fallback. */
protected def assertCodegenRan(f: => Unit): Unit = {
protected def assertCodegenRan[T](f: => T): T = {
CometScalaUDFCodegen.resetStats()
f
val result = f
val after = CometScalaUDFCodegen.stats()
assert(
after.compileCount + after.cacheHitCount >= 1,
s"expected codegen dispatcher activity, got $after")
result
}

/** Asserts the dispatcher did not run during `f`, guarding a native-path control case. */
protected def assertCodegenDidNotRun[T](f: => T): T = {
CometScalaUDFCodegen.resetStats()
val result = f
val after = CometScalaUDFCodegen.stats()
assert(after.totalLookups == 0, s"expected no codegen dispatcher activity, got $after")
result
}

/**
Expand Down
Loading