Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ For applicable cases that are not selected for native execution automatically, `
The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:

- Not all input data types are supported; falls back to Spark for unsupported types
- Comet has no native path for a nondeterministic operand such as `rand()` or `monotonically_increasing_id()`, because the native `NULL` guard would evaluate it twice.

## Sequence

Expand All @@ -110,4 +111,10 @@ The following cases are not supported by Comet and always fall back to Spark, re

- shuffle requires a resolved random seed

## Size

The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:

- Comet has no native path for a nondeterministic operand such as `rand()` or `monotonically_increasing_id()`, because the native `NULL` guard would evaluate it twice.

<!--END:EXPR_COMPAT-->
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ For applicable cases that are not selected for native execution automatically, `
The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:

- Not all input data types are supported; falls back to Spark for unsupported types
- Comet has no native path for a nondeterministic operand such as `rand()` or `monotonically_increasing_id()`, because the native `NULL` guard would evaluate it twice.

## Sequence

Expand All @@ -110,4 +111,10 @@ The following cases are not supported by Comet and always fall back to Spark, re

- shuffle requires a resolved random seed

## Size

The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:

- Comet has no native path for a nondeterministic operand such as `rand()` or `monotonically_increasing_id()`, because the native `NULL` guard would evaluate it twice.

<!--END:EXPR_COMPAT-->
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ For applicable cases that are not selected for native execution automatically, `
The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:

- Not all input data types are supported; falls back to Spark for unsupported types
- Comet has no native path for a nondeterministic operand such as `rand()` or `monotonically_increasing_id()`, because the native `NULL` guard would evaluate it twice.

## Sequence

Expand All @@ -110,4 +111,10 @@ The following cases are not supported by Comet and always fall back to Spark, re

- shuffle requires a resolved random seed

## Size

The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:

- Comet has no native path for a nondeterministic operand such as `rand()` or `monotonically_increasing_id()`, because the native `NULL` guard would evaluate it twice.

<!--END:EXPR_COMPAT-->
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ For applicable cases that are not selected for native execution automatically, `
The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:

- Not all input data types are supported; falls back to Spark for unsupported types
- Comet has no native path for a nondeterministic operand such as `rand()` or `monotonically_increasing_id()`, because the native `NULL` guard would evaluate it twice.

## Sequence

Expand All @@ -110,4 +111,10 @@ The following cases are not supported by Comet and always fall back to Spark, re

- shuffle requires a resolved random seed

## Size

The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:

- Comet has no native path for a nondeterministic operand such as `rand()` or `monotonically_increasing_id()`, because the native `NULL` guard would evaluate it twice.

<!--END:EXPR_COMPAT-->
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ For applicable cases that are not selected for native execution automatically, `
The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:

- Not all input data types are supported; falls back to Spark for unsupported types
- Comet has no native path for a nondeterministic operand such as `rand()` or `monotonically_increasing_id()`, because the native `NULL` guard would evaluate it twice.

## Sequence

Expand All @@ -110,4 +111,10 @@ The following cases are not supported by Comet and always fall back to Spark, re

- shuffle requires a resolved random seed

## Size

The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:

- Comet has no native path for a nondeterministic operand such as `rand()` or `monotonically_increasing_id()`, because the native `NULL` guard would evaluate it twice.

<!--END:EXPR_COMPAT-->
19 changes: 18 additions & 1 deletion spark/src/main/scala/org/apache/comet/serde/arrays.scala
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,13 @@ import org.apache.comet.expressions.{CometCast, CometEvalMode}
import org.apache.comet.serde.QueryPlanSerde._
import org.apache.comet.shims.{CometExprShim, CometTypeShim}

private[serde] object NullGuardSupport {

val nondeterministicReason: String =
"Comet has no native path for a nondeterministic operand such as `rand()` or " +
"`monotonically_increasing_id()`, because the native `NULL` guard would evaluate it twice."
}

object CometArrayRemove
extends CometExpressionSerde[ArrayRemove]
with CometExprShim
Expand Down Expand Up @@ -1060,7 +1067,13 @@ object CometArrayFilter extends CometExpressionSerde[ArrayFilter] {

object CometSize extends CometExpressionSerde[Size] {

override def getUnsupportedReasons(): Seq[String] =
Seq(NullGuardSupport.nondeterministicReason)

override def getSupportLevel(expr: Size): SupportLevel = {
if (!expr.child.deterministic) {
return Unsupported(Some(NullGuardSupport.nondeterministicReason))
}
expr.child.dataType match {
case _: ArrayType => Compatible()
case _: MapType => Compatible()
Expand Down Expand Up @@ -1139,7 +1152,8 @@ object CometArrayPosition extends CometExpressionSerde[ArrayPosition] with Array
object CometArraysZip extends CometExpressionSerde[ArraysZip] {

override def getUnsupportedReasons(): Seq[String] = Seq(
"Not all input data types are supported; falls back to Spark for unsupported types")
"Not all input data types are supported; falls back to Spark for unsupported types",
NullGuardSupport.nondeterministicReason)

private def isTypeSupported(dt: DataType): Boolean = {
import DataTypes._
Expand All @@ -1155,6 +1169,9 @@ object CometArraysZip extends CometExpressionSerde[ArraysZip] {
}

override def getSupportLevel(expr: ArraysZip): SupportLevel = {
if (expr.children.exists(!_.deterministic)) {
return Unsupported(Some(NullGuardSupport.nondeterministicReason))
}
val inputTypes = expr.children.map(_.dataType).toSet
for (dt <- inputTypes) {
if (!isTypeSupported(dt)) {
Expand Down
7 changes: 7 additions & 0 deletions spark/src/main/scala/org/apache/comet/serde/maps.scala
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types._

import org.apache.comet.CometSparkSessionExtensions.withFallbackReason
import org.apache.comet.DataTypeSupport.isComplexType
import org.apache.comet.serde.QueryPlanSerde.{exprToProtoInternal, hasNonDefaultStringCollation, scalarFunctionExprToProto, withNullShortCircuit}
import org.apache.comet.shims.CometTypeShim
Expand Down Expand Up @@ -193,6 +194,12 @@ object CometMapFromArrays
expr: MapFromArrays,
inputs: Seq[Attribute],
binding: Boolean): Option[ExprOuterClass.Expr] = {
// Declined in convert rather than as Unsupported, which would route it to the codegen
// dispatcher instead of falling back to Spark.
if (expr.children.exists(!_.deterministic)) {
withFallbackReason(expr, NullGuardSupport.nondeterministicReason)
return None
}
val keysExpr = exprToProtoInternal(expr.left, inputs, binding)
val valuesExpr = exprToProtoInternal(expr.right, inputs, binding)
val keyType = expr.left.dataType.asInstanceOf[ArrayType].elementType
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
-- Licensed to the Apache Software Foundation (ASF) under one
-- or more contributor license agreements. See the NOTICE file
-- distributed with this work for additional information
-- regarding copyright ownership. The ASF licenses this file
-- to you under the Apache License, Version 2.0 (the
-- "License"); you may not use this file except in compliance
-- with the License. You may obtain a copy of the License at
--
-- http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing,
-- software distributed under the License is distributed on an
-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-- KIND, either express or implied. See the License for the
-- specific language governing permissions and limitations
-- under the License.

-- The native NULL guard serializes each child twice, so a nondeterministic child would drift
-- between the copies and falls back to Spark instead.

statement
CREATE TABLE test_arrays_zip_nondet(_1 int, arr array<int>) USING parquet

statement
INSERT INTO test_arrays_zip_nondet
SELECT id, IF(id % 4 = 3, NULL, array(id, id + 1)) FROM range(0, 16)

query expect_fallback(because the native `NULL` guard would evaluate it twice)
SELECT _1, arrays_zip(IF(monotonically_increasing_id() % 2 = 0, array(1), CAST(NULL AS ARRAY<INT>)), array(2)) AS z
FROM test_arrays_zip_nondet

query expect_fallback(because the native `NULL` guard would evaluate it twice)
SELECT _1, arrays_zip(array(2), IF(monotonically_increasing_id() % 2 = 0, array(1), CAST(NULL AS ARRAY<INT>))) AS z
FROM test_arrays_zip_nondet

-- Declined even when non-nullable, rather than relying on the guard matching every row
query expect_fallback(because the native `NULL` guard would evaluate it twice)
SELECT _1, arrays_zip(array(monotonically_increasing_id()), array(2)) AS z
FROM test_arrays_zip_nondet

-- Every fourth row brings a NULL array from the input rather than from the IF
query expect_fallback(because the native `NULL` guard would evaluate it twice)
SELECT _1, arrays_zip(IF(monotonically_increasing_id() % 2 = 0, arr, CAST(NULL AS ARRAY<INT>)), arr) AS z
FROM test_arrays_zip_nondet

query expect_native(arrays_zip)
SELECT _1, arrays_zip(IF(_1 % 2 = 0, array(1), CAST(NULL AS ARRAY<INT>)), array(2)) AS z
FROM test_arrays_zip_nondet
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
-- Licensed to the Apache Software Foundation (ASF) under one
-- or more contributor license agreements. See the NOTICE file
-- distributed with this work for additional information
-- regarding copyright ownership. The ASF licenses this file
-- to you under the Apache License, Version 2.0 (the
-- "License"); you may not use this file except in compliance
-- with the License. You may obtain a copy of the License at
--
-- http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing,
-- software distributed under the License is distributed on an
-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-- KIND, either express or implied. See the License for the
-- specific language governing permissions and limitations
-- under the License.

-- The native NULL guard serializes the child twice, so a nondeterministic child would drift
-- between the copies and falls back to Spark instead.

-- ConfigMatrix: spark.sql.legacy.sizeOfNull=true,false

statement
CREATE TABLE test_size_nondet(_1 int, arr array<int>) USING parquet

statement
INSERT INTO test_size_nondet
SELECT id, IF(id % 4 = 3, NULL, array(id, id + 1)) FROM range(0, 16)

query expect_fallback(because the native `NULL` guard would evaluate it twice)
SELECT _1, size(IF(monotonically_increasing_id() % 2 = 0, array(1), CAST(NULL AS ARRAY<INT>))) AS s
FROM test_size_nondet

-- Declined even when non-nullable, rather than relying on the guard matching every row
query expect_fallback(because the native `NULL` guard would evaluate it twice)
SELECT _1, size(array(monotonically_increasing_id())) AS s
FROM test_size_nondet

query expect_native(size)
SELECT _1, size(IF(_1 % 2 = 0, array(1), CAST(NULL AS ARRAY<INT>))) AS s
FROM test_size_nondet

-- Every fourth row brings a NULL array from the input rather than from the IF
query expect_fallback(because the native `NULL` guard would evaluate it twice)
SELECT _1, size(IF(monotonically_increasing_id() % 2 = 0, arr, CAST(NULL AS ARRAY<INT>))) AS s
FROM test_size_nondet
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
-- Licensed to the Apache Software Foundation (ASF) under one
-- or more contributor license agreements. See the NOTICE file
-- distributed with this work for additional information
-- regarding copyright ownership. The ASF licenses this file
-- to you under the Apache License, Version 2.0 (the
-- "License"); you may not use this file except in compliance
-- with the License. You may obtain a copy of the License at
--
-- http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing,
-- software distributed under the License is distributed on an
-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-- KIND, either express or implied. See the License for the
-- specific language governing permissions and limitations
-- under the License.

-- The native NULL guard serializes both children twice, so a nondeterministic child would drift
-- between the copies and falls back to Spark instead.

statement
CREATE TABLE test_map_from_arrays_nondet(_1 int, k array<int>, v array<int>) USING parquet

statement
INSERT INTO test_map_from_arrays_nondet
SELECT id, IF(id % 4 = 3, NULL, array(id, id + 100)), array(id * 2, id * 2 + 1) FROM range(0, 16)

query expect_fallback(because the native `NULL` guard would evaluate it twice)
SELECT _1, map_from_arrays(IF(monotonically_increasing_id() % 2 = 0, array(1), CAST(NULL AS ARRAY<INT>)), array(2)) AS m
FROM test_map_from_arrays_nondet

query expect_fallback(because the native `NULL` guard would evaluate it twice)
SELECT _1, map_from_arrays(array(1), IF(monotonically_increasing_id() % 2 = 0, array(2), CAST(NULL AS ARRAY<INT>))) AS m
FROM test_map_from_arrays_nondet

-- Declined even when non-nullable, rather than relying on the guard matching every row
query expect_fallback(because the native `NULL` guard would evaluate it twice)
SELECT _1, map_from_arrays(array(monotonically_increasing_id()), array(2)) AS m
FROM test_map_from_arrays_nondet

-- Every fourth row brings a NULL keys array from the input rather than from the IF
query expect_fallback(because the native `NULL` guard would evaluate it twice)
SELECT _1, map_from_arrays(IF(monotonically_increasing_id() % 2 = 0, k, CAST(NULL AS ARRAY<INT>)), v) AS m
FROM test_map_from_arrays_nondet

query expect_native(map_from_arrays)
SELECT _1, map_from_arrays(IF(_1 % 2 = 0, array(1), CAST(NULL AS ARRAY<INT>)), array(2)) AS m
FROM test_map_from_arrays_nondet

-- DOUBLE keys are Incompatible, so by default the codegen dispatcher runs Spark's own code
query expect_dispatch(map_from_arrays)
SELECT _1, map_from_arrays(IF(monotonically_increasing_id() % 2 = 0, array(CAST(_1 AS DOUBLE)), CAST(NULL AS ARRAY<DOUBLE>)), array(2)) AS m
FROM test_map_from_arrays_nondet

statement
SET spark.comet.expression.MapFromArrays.allowIncompatible=true

-- The native opt-in reaches convert directly, which still declines the nondeterministic child
query expect_fallback(because the native `NULL` guard would evaluate it twice)
SELECT _1, map_from_arrays(IF(monotonically_increasing_id() % 2 = 0, array(CAST(_1 AS DOUBLE)), CAST(NULL AS ARRAY<DOUBLE>)), array(2)) AS m
FROM test_map_from_arrays_nondet