diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/array.md b/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/array.md index 3e94336b308..691f1761574 100644 --- a/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/array.md +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/array.md @@ -92,6 +92,7 @@ For applicable cases that are not selected for native execution automatically, ` The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: - Not all input data types are supported; falls back to Spark for unsupported types +- Comet has no native path for a nondeterministic operand such as `rand()` or `monotonically_increasing_id()`, because the native `NULL` guard would evaluate it twice. ## Sequence @@ -110,4 +111,10 @@ The following cases are not supported by Comet and always fall back to Spark, re - shuffle requires a resolved random seed +## Size + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Comet has no native path for a nondeterministic operand such as `rand()` or `monotonically_increasing_id()`, because the native `NULL` guard would evaluate it twice. + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/array.md b/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/array.md index 3e94336b308..691f1761574 100644 --- a/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/array.md +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/array.md @@ -92,6 +92,7 @@ For applicable cases that are not selected for native execution automatically, ` The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: - Not all input data types are supported; falls back to Spark for unsupported types +- Comet has no native path for a nondeterministic operand such as `rand()` or `monotonically_increasing_id()`, because the native `NULL` guard would evaluate it twice. ## Sequence @@ -110,4 +111,10 @@ The following cases are not supported by Comet and always fall back to Spark, re - shuffle requires a resolved random seed +## Size + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Comet has no native path for a nondeterministic operand such as `rand()` or `monotonically_increasing_id()`, because the native `NULL` guard would evaluate it twice. + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/array.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/array.md index 3e94336b308..691f1761574 100644 --- a/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/array.md +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/array.md @@ -92,6 +92,7 @@ For applicable cases that are not selected for native execution automatically, ` The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: - Not all input data types are supported; falls back to Spark for unsupported types +- Comet has no native path for a nondeterministic operand such as `rand()` or `monotonically_increasing_id()`, because the native `NULL` guard would evaluate it twice. ## Sequence @@ -110,4 +111,10 @@ The following cases are not supported by Comet and always fall back to Spark, re - shuffle requires a resolved random seed +## Size + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Comet has no native path for a nondeterministic operand such as `rand()` or `monotonically_increasing_id()`, because the native `NULL` guard would evaluate it twice. + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/array.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/array.md index 3e94336b308..691f1761574 100644 --- a/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/array.md +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/array.md @@ -92,6 +92,7 @@ For applicable cases that are not selected for native execution automatically, ` The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: - Not all input data types are supported; falls back to Spark for unsupported types +- Comet has no native path for a nondeterministic operand such as `rand()` or `monotonically_increasing_id()`, because the native `NULL` guard would evaluate it twice. ## Sequence @@ -110,4 +111,10 @@ The following cases are not supported by Comet and always fall back to Spark, re - shuffle requires a resolved random seed +## Size + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Comet has no native path for a nondeterministic operand such as `rand()` or `monotonically_increasing_id()`, because the native `NULL` guard would evaluate it twice. + diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/array.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/array.md index 3e94336b308..691f1761574 100644 --- a/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/array.md +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/array.md @@ -92,6 +92,7 @@ For applicable cases that are not selected for native execution automatically, ` The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: - Not all input data types are supported; falls back to Spark for unsupported types +- Comet has no native path for a nondeterministic operand such as `rand()` or `monotonically_increasing_id()`, because the native `NULL` guard would evaluate it twice. ## Sequence @@ -110,4 +111,10 @@ The following cases are not supported by Comet and always fall back to Spark, re - shuffle requires a resolved random seed +## Size + +The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting: + +- Comet has no native path for a nondeterministic operand such as `rand()` or `monotonically_increasing_id()`, because the native `NULL` guard would evaluate it twice. + diff --git a/spark/src/main/scala/org/apache/comet/serde/arrays.scala b/spark/src/main/scala/org/apache/comet/serde/arrays.scala index 3df09ddf874..edd1638ac30 100644 --- a/spark/src/main/scala/org/apache/comet/serde/arrays.scala +++ b/spark/src/main/scala/org/apache/comet/serde/arrays.scala @@ -35,6 +35,13 @@ import org.apache.comet.expressions.{CometCast, CometEvalMode} import org.apache.comet.serde.QueryPlanSerde._ import org.apache.comet.shims.{CometExprShim, CometTypeShim} +private[serde] object NullGuardSupport { + + val nondeterministicReason: String = + "Comet has no native path for a nondeterministic operand such as `rand()` or " + + "`monotonically_increasing_id()`, because the native `NULL` guard would evaluate it twice." +} + object CometArrayRemove extends CometExpressionSerde[ArrayRemove] with CometExprShim @@ -1060,7 +1067,13 @@ object CometArrayFilter extends CometExpressionSerde[ArrayFilter] { object CometSize extends CometExpressionSerde[Size] { + override def getUnsupportedReasons(): Seq[String] = + Seq(NullGuardSupport.nondeterministicReason) + override def getSupportLevel(expr: Size): SupportLevel = { + if (!expr.child.deterministic) { + return Unsupported(Some(NullGuardSupport.nondeterministicReason)) + } expr.child.dataType match { case _: ArrayType => Compatible() case _: MapType => Compatible() @@ -1139,7 +1152,8 @@ object CometArrayPosition extends CometExpressionSerde[ArrayPosition] with Array object CometArraysZip extends CometExpressionSerde[ArraysZip] { override def getUnsupportedReasons(): Seq[String] = Seq( - "Not all input data types are supported; falls back to Spark for unsupported types") + "Not all input data types are supported; falls back to Spark for unsupported types", + NullGuardSupport.nondeterministicReason) private def isTypeSupported(dt: DataType): Boolean = { import DataTypes._ @@ -1155,6 +1169,9 @@ object CometArraysZip extends CometExpressionSerde[ArraysZip] { } override def getSupportLevel(expr: ArraysZip): SupportLevel = { + if (expr.children.exists(!_.deterministic)) { + return Unsupported(Some(NullGuardSupport.nondeterministicReason)) + } val inputTypes = expr.children.map(_.dataType).toSet for (dt <- inputTypes) { if (!isTypeSupported(dt)) { diff --git a/spark/src/main/scala/org/apache/comet/serde/maps.scala b/spark/src/main/scala/org/apache/comet/serde/maps.scala index 86376ccfca8..da9f076034d 100644 --- a/spark/src/main/scala/org/apache/comet/serde/maps.scala +++ b/spark/src/main/scala/org/apache/comet/serde/maps.scala @@ -23,6 +23,7 @@ import org.apache.spark.sql.catalyst.expressions._ import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.types._ +import org.apache.comet.CometSparkSessionExtensions.withFallbackReason import org.apache.comet.DataTypeSupport.isComplexType import org.apache.comet.serde.QueryPlanSerde.{exprToProtoInternal, hasNonDefaultStringCollation, scalarFunctionExprToProto, withNullShortCircuit} import org.apache.comet.shims.CometTypeShim @@ -193,6 +194,12 @@ object CometMapFromArrays expr: MapFromArrays, inputs: Seq[Attribute], binding: Boolean): Option[ExprOuterClass.Expr] = { + // Declined in convert rather than as Unsupported, which would route it to the codegen + // dispatcher instead of falling back to Spark. + if (expr.children.exists(!_.deterministic)) { + withFallbackReason(expr, NullGuardSupport.nondeterministicReason) + return None + } val keysExpr = exprToProtoInternal(expr.left, inputs, binding) val valuesExpr = exprToProtoInternal(expr.right, inputs, binding) val keyType = expr.left.dataType.asInstanceOf[ArrayType].elementType diff --git a/spark/src/test/resources/sql-tests/expressions/array/arrays_zip_nondeterministic_child.sql b/spark/src/test/resources/sql-tests/expressions/array/arrays_zip_nondeterministic_child.sql new file mode 100644 index 00000000000..1b5c46aa754 --- /dev/null +++ b/spark/src/test/resources/sql-tests/expressions/array/arrays_zip_nondeterministic_child.sql @@ -0,0 +1,48 @@ +-- Licensed to the Apache Software Foundation (ASF) under one +-- or more contributor license agreements. See the NOTICE file +-- distributed with this work for additional information +-- regarding copyright ownership. The ASF licenses this file +-- to you under the Apache License, Version 2.0 (the +-- "License"); you may not use this file except in compliance +-- with the License. You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, +-- software distributed under the License is distributed on an +-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +-- KIND, either express or implied. See the License for the +-- specific language governing permissions and limitations +-- under the License. + +-- The native NULL guard serializes each child twice, so a nondeterministic child would drift +-- between the copies and falls back to Spark instead. + +statement +CREATE TABLE test_arrays_zip_nondet(_1 int, arr array) USING parquet + +statement +INSERT INTO test_arrays_zip_nondet +SELECT id, IF(id % 4 = 3, NULL, array(id, id + 1)) FROM range(0, 16) + +query expect_fallback(because the native `NULL` guard would evaluate it twice) +SELECT _1, arrays_zip(IF(monotonically_increasing_id() % 2 = 0, array(1), CAST(NULL AS ARRAY)), array(2)) AS z +FROM test_arrays_zip_nondet + +query expect_fallback(because the native `NULL` guard would evaluate it twice) +SELECT _1, arrays_zip(array(2), IF(monotonically_increasing_id() % 2 = 0, array(1), CAST(NULL AS ARRAY))) AS z +FROM test_arrays_zip_nondet + +-- Declined even when non-nullable, rather than relying on the guard matching every row +query expect_fallback(because the native `NULL` guard would evaluate it twice) +SELECT _1, arrays_zip(array(monotonically_increasing_id()), array(2)) AS z +FROM test_arrays_zip_nondet + +-- Every fourth row brings a NULL array from the input rather than from the IF +query expect_fallback(because the native `NULL` guard would evaluate it twice) +SELECT _1, arrays_zip(IF(monotonically_increasing_id() % 2 = 0, arr, CAST(NULL AS ARRAY)), arr) AS z +FROM test_arrays_zip_nondet + +query expect_native(arrays_zip) +SELECT _1, arrays_zip(IF(_1 % 2 = 0, array(1), CAST(NULL AS ARRAY)), array(2)) AS z +FROM test_arrays_zip_nondet diff --git a/spark/src/test/resources/sql-tests/expressions/array/size_nondeterministic_child.sql b/spark/src/test/resources/sql-tests/expressions/array/size_nondeterministic_child.sql new file mode 100644 index 00000000000..027b78f13e9 --- /dev/null +++ b/spark/src/test/resources/sql-tests/expressions/array/size_nondeterministic_child.sql @@ -0,0 +1,46 @@ +-- Licensed to the Apache Software Foundation (ASF) under one +-- or more contributor license agreements. See the NOTICE file +-- distributed with this work for additional information +-- regarding copyright ownership. The ASF licenses this file +-- to you under the Apache License, Version 2.0 (the +-- "License"); you may not use this file except in compliance +-- with the License. You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, +-- software distributed under the License is distributed on an +-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +-- KIND, either express or implied. See the License for the +-- specific language governing permissions and limitations +-- under the License. + +-- The native NULL guard serializes the child twice, so a nondeterministic child would drift +-- between the copies and falls back to Spark instead. + +-- ConfigMatrix: spark.sql.legacy.sizeOfNull=true,false + +statement +CREATE TABLE test_size_nondet(_1 int, arr array) USING parquet + +statement +INSERT INTO test_size_nondet +SELECT id, IF(id % 4 = 3, NULL, array(id, id + 1)) FROM range(0, 16) + +query expect_fallback(because the native `NULL` guard would evaluate it twice) +SELECT _1, size(IF(monotonically_increasing_id() % 2 = 0, array(1), CAST(NULL AS ARRAY))) AS s +FROM test_size_nondet + +-- Declined even when non-nullable, rather than relying on the guard matching every row +query expect_fallback(because the native `NULL` guard would evaluate it twice) +SELECT _1, size(array(monotonically_increasing_id())) AS s +FROM test_size_nondet + +query expect_native(size) +SELECT _1, size(IF(_1 % 2 = 0, array(1), CAST(NULL AS ARRAY))) AS s +FROM test_size_nondet + +-- Every fourth row brings a NULL array from the input rather than from the IF +query expect_fallback(because the native `NULL` guard would evaluate it twice) +SELECT _1, size(IF(monotonically_increasing_id() % 2 = 0, arr, CAST(NULL AS ARRAY))) AS s +FROM test_size_nondet diff --git a/spark/src/test/resources/sql-tests/expressions/map/map_from_arrays_nondeterministic_child.sql b/spark/src/test/resources/sql-tests/expressions/map/map_from_arrays_nondeterministic_child.sql new file mode 100644 index 00000000000..d3978a7e770 --- /dev/null +++ b/spark/src/test/resources/sql-tests/expressions/map/map_from_arrays_nondeterministic_child.sql @@ -0,0 +1,61 @@ +-- Licensed to the Apache Software Foundation (ASF) under one +-- or more contributor license agreements. See the NOTICE file +-- distributed with this work for additional information +-- regarding copyright ownership. The ASF licenses this file +-- to you under the Apache License, Version 2.0 (the +-- "License"); you may not use this file except in compliance +-- with the License. You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, +-- software distributed under the License is distributed on an +-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +-- KIND, either express or implied. See the License for the +-- specific language governing permissions and limitations +-- under the License. + +-- The native NULL guard serializes both children twice, so a nondeterministic child would drift +-- between the copies and falls back to Spark instead. + +statement +CREATE TABLE test_map_from_arrays_nondet(_1 int, k array, v array) USING parquet + +statement +INSERT INTO test_map_from_arrays_nondet +SELECT id, IF(id % 4 = 3, NULL, array(id, id + 100)), array(id * 2, id * 2 + 1) FROM range(0, 16) + +query expect_fallback(because the native `NULL` guard would evaluate it twice) +SELECT _1, map_from_arrays(IF(monotonically_increasing_id() % 2 = 0, array(1), CAST(NULL AS ARRAY)), array(2)) AS m +FROM test_map_from_arrays_nondet + +query expect_fallback(because the native `NULL` guard would evaluate it twice) +SELECT _1, map_from_arrays(array(1), IF(monotonically_increasing_id() % 2 = 0, array(2), CAST(NULL AS ARRAY))) AS m +FROM test_map_from_arrays_nondet + +-- Declined even when non-nullable, rather than relying on the guard matching every row +query expect_fallback(because the native `NULL` guard would evaluate it twice) +SELECT _1, map_from_arrays(array(monotonically_increasing_id()), array(2)) AS m +FROM test_map_from_arrays_nondet + +-- Every fourth row brings a NULL keys array from the input rather than from the IF +query expect_fallback(because the native `NULL` guard would evaluate it twice) +SELECT _1, map_from_arrays(IF(monotonically_increasing_id() % 2 = 0, k, CAST(NULL AS ARRAY)), v) AS m +FROM test_map_from_arrays_nondet + +query expect_native(map_from_arrays) +SELECT _1, map_from_arrays(IF(_1 % 2 = 0, array(1), CAST(NULL AS ARRAY)), array(2)) AS m +FROM test_map_from_arrays_nondet + +-- DOUBLE keys are Incompatible, so by default the codegen dispatcher runs Spark's own code +query expect_dispatch(map_from_arrays) +SELECT _1, map_from_arrays(IF(monotonically_increasing_id() % 2 = 0, array(CAST(_1 AS DOUBLE)), CAST(NULL AS ARRAY)), array(2)) AS m +FROM test_map_from_arrays_nondet + +statement +SET spark.comet.expression.MapFromArrays.allowIncompatible=true + +-- The native opt-in reaches convert directly, which still declines the nondeterministic child +query expect_fallback(because the native `NULL` guard would evaluate it twice) +SELECT _1, map_from_arrays(IF(monotonically_increasing_id() % 2 = 0, array(CAST(_1 AS DOUBLE)), CAST(NULL AS ARRAY)), array(2)) AS m +FROM test_map_from_arrays_nondet