Repository navigation
Conversation
Add SparkNullability, which declares a function's nullability as Spark's nullable and nullIntolerant members do. It provides the return field and, for a null-intolerant function, a simplify that replaces a call with a NULL literal argument with a NULL literal, as Spark's NullPropagation does. Move ascii, char, length, crc32, sha1, date_add, date_sub, bit_get, substring and slice onto it. Part of apache#383.
…ullability Declare Spark's nullable and nullIntolerant for every function that is null-intolerant in Spark, so that a call with a NULL literal argument folds to a NULL literal and the result field follows Spark's nullability. Functions that only implemented return_type were always nullable. Those whose Spark expression takes nullability from its children are now non-nullable for non-nullable arguments: array_position, array_remove, atan2, contains, hex, hour, minute, second, hypot, negative, pow, soundex and space. floor, make_valid_utf8 and the functions Spark marks as always nullable are now always nullable. make_dt_interval is always nullable, because an overflow returns NULL (apache#20). This fixes apache#315, so its records no longer skip evaluate mode. Part of apache#383.
…ith a macro Add spark_nullability!, which implements simplify, and with return_type also return_field_from_args, from a function's SparkNullability. This replaces the same eight-line simplify, and the three-line return_field_from_args, in each function. Part of apache#383.
The LIKE operator plans as DataFusion's built-in, so the like record now calls the like function only. Part of apache#383.
- Make SparkNullability an enum of the two nullable rules, since every function that uses it is null-intolerant, and drop the unused flag. - Take &dyn ScalarUDFImpl, so the helpers are compiled once. - Remove the NULL-folding prologue from add_months, date_diff and unix_date: their rewrites already fold a NULL argument. - Share list_element_field from functions_nested_utils and use it in array_contains. - Move unix_seconds, unix_millis and unix_micros onto SparkNullability. Part of apache#383.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
make_dt_intervaldeclares a non-nullable result but can return NULL #315.nullableandnullIntolerantper function in one place #383.Rationale for this change
Spark declares two nullability properties on each expression:
nullable(whether the result can be NULL) andnullIntolerant(a NULL input always gives a NULL output, whichNullPropagationuses to foldf(col, NULL)to NULL). Here each function worked out its nullability by hand, so:return_typewere always nullable, even where Spark takes nullability from the children.floor,make_valid_utf8).make_dt_intervaldeclared a non-nullable result but returns NULL on overflow, so a batch failed validation when the call was not constant-folded ([Bug]make_dt_intervaldeclares a non-nullable result but can return NULL #315).pmod(col, NULL), was not folded to NULL at plan time.What changes are included in this PR?
function/nullability.rsaddsSparkNullability, for a null-intolerant function. Its two values say whether the result is nullable when an argument is, or always. It provides:return_field_from_argssimplifythat replaces a call with a NULL literal argument with a typed NULL literalspark_nullability!macro implementssimplifyfrom it insideimpl ScalarUDFImpl.spark_nullability!(NULLABILITY, return_type)also implementsreturn_field_from_argsfromreturn_type.NULLABILITYconstant whose doc comment names the Spark expression and its rule, so the declaration can be checked against Spark's source.add_months,date_diffandunix_datekeep their ownsimplify, whose rewrite already folds a NULL argument.array_position,array_remove,atan2,contains,hex,hour,minute,second,hypot,negative,pow,soundexandspace. I checked that each only returns NULL for a NULL argument.floor,make_valid_utf8andmake_dt_intervalare now always nullable.make_dt_intervaldiffers from Spark here, because an overflow returns NULL instead of raising an error (Spark make_dt_interval/make_interval return NULL on overflow instead of erroring #20).make_dateandmake_intervalstay always nullable. With ANSI mode on, Spark makes them nullable from their arguments, butreturn_field_from_argscannot see ANSI mode.array_containsis also nullable when the array can hold NULL elements, andmap_from_entrieswhen an entry can be NULL, as in Spark.list_element_fieldmoves fromarray_insert.rstofunctions_nested_utils.rsso thatarray_containscan share it.Not included:
date_partand the functions that Spark replaces with a null-intolerantStaticInvoke(base64,url_encode, ...), which belong with #386, andspark_cast(#88).Are these changes tested?
SparkNullabilityinnullability.rs, and for thearray_containsrule..sltrecords for a column combined with a NULL argument, instring/substring.slt,array/slice.slt,datetime/date_add.slt,datetime/date_sub.slt,bitwise/bit_get.slt,string/contains.slt,predicate/like.slt,math/pmod.sltandmath/atan2.slt. They run in bothsimplifyandevaluatemode, and anEXPLAINrecord insubstring.sltshows the fold.make_dt_interval.sltrecords that were skipped inevaluatemode for [Bug]make_dt_intervaldeclares a non-nullable result but can return NULL #315 now run in both modes..sltfiles against Spark 4.2.0 with the PySpark runner, which found no differences.Are there any user-facing changes?
make_dt_intervalno longer fails inevaluatemode when it overflows.