Skip to content

feat(spark): declare Spark's nullable and nullIntolerant with SparkNullability - #389

Open
andygrove wants to merge 5 commits into
apache:mainfrom
andygrove:null-intolerant
Open

andygrove wants to merge 5 commits into
apache:mainfrom
andygrove:null-intolerant

Conversation

@andygrove

@andygrove andygrove commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

Spark declares two nullability properties on each expression: nullable (whether the result can be NULL) and nullIntolerant (a NULL input always gives a NULL output, which NullPropagation uses to fold f(col, NULL) to NULL). Here each function worked out its nullability by hand, so:

  • Functions that only implemented return_type were always nullable, even where Spark takes nullability from the children.
  • Some followed their arguments where Spark is always nullable (floor, make_valid_utf8).
  • make_dt_interval declared a non-nullable result but returns NULL on overflow, so a batch failed validation when the call was not constant-folded ([Bug] make_dt_interval declares a non-nullable result but can return NULL #315).
  • A call with a NULL literal argument and a column argument, such as pmod(col, NULL), was not folded to NULL at plan time.

What changes are included in this PR?

  • function/nullability.rs adds SparkNullability, for a null-intolerant function. Its two values say whether the result is nullable when an argument is, or always. It provides:
    • the return field, for return_field_from_args
    • a simplify that replaces a call with a NULL literal argument with a typed NULL literal
  • The spark_nullability! macro implements simplify from it inside impl ScalarUDFImpl. spark_nullability!(NULLABILITY, return_type) also implements return_field_from_args from return_type.
  • About 60 functions that Spark marks as null-intolerant now use it. Each declares a NULLABILITY constant whose doc comment names the Spark expression and its rule, so the declaration can be checked against Spark's source.
  • add_months, date_diff and unix_date keep their own simplify, whose rewrite already folds a NULL argument.
  • These are now non-nullable for non-nullable arguments, as in Spark: array_position, array_remove, atan2, contains, hex, hour, minute, second, hypot, negative, pow, soundex and space. I checked that each only returns NULL for a NULL argument.
  • floor, make_valid_utf8 and make_dt_interval are now always nullable. make_dt_interval differs from Spark here, because an overflow returns NULL instead of raising an error (Spark make_dt_interval/make_interval return NULL on overflow instead of erroring #20).
  • make_date and make_interval stay always nullable. With ANSI mode on, Spark makes them nullable from their arguments, but return_field_from_args cannot see ANSI mode.
  • array_contains is also nullable when the array can hold NULL elements, and map_from_entries when an entry can be NULL, as in Spark. list_element_field moves from array_insert.rs to functions_nested_utils.rs so that array_contains can share it.

Not included: date_part and the functions that Spark replaces with a null-intolerant StaticInvoke (base64, url_encode, ...), which belong with #386, and spark_cast (#88).

Are these changes tested?

  • Unit tests for SparkNullability in nullability.rs, and for the array_contains rule.
  • New .slt records for a column combined with a NULL argument, in string/substring.slt, array/slice.slt, datetime/date_add.slt, datetime/date_sub.slt, bitwise/bit_get.slt, string/contains.slt, predicate/like.slt, math/pmod.slt and math/atan2.slt. They run in both simplify and evaluate mode, and an EXPLAIN record in substring.slt shows the fold.
  • The make_dt_interval.slt records that were skipped in evaluate mode for [Bug] make_dt_interval declares a non-nullable result but can return NULL #315 now run in both modes.
  • I checked the changed .slt files against Spark 4.2.0 with the PySpark runner, which found no differences.

Are there any user-facing changes?

  • A call to a null-intolerant function with a NULL literal argument is folded to NULL during planning.
  • The nullability of the result fields above changes to match Spark. make_dt_interval no longer fails in evaluate mode when it overflows.

Add SparkNullability, which declares a function's nullability as Spark's
nullable and nullIntolerant members do. It provides the return field and,
for a null-intolerant function, a simplify that replaces a call with a NULL
literal argument with a NULL literal, as Spark's NullPropagation does.

Move ascii, char, length, crc32, sha1, date_add, date_sub, bit_get,
substring and slice onto it.

Part of apache#383.
…ullability

Declare Spark's nullable and nullIntolerant for every function that is
null-intolerant in Spark, so that a call with a NULL literal argument folds
to a NULL literal and the result field follows Spark's nullability.

Functions that only implemented return_type were always nullable. Those
whose Spark expression takes nullability from its children are now
non-nullable for non-nullable arguments: array_position, array_remove,
atan2, contains, hex, hour, minute, second, hypot, negative, pow, soundex
and space. floor, make_valid_utf8 and the functions Spark marks as always
nullable are now always nullable.

make_dt_interval is always nullable, because an overflow returns NULL
(apache#20). This fixes apache#315, so its records no longer skip evaluate mode.

Part of apache#383.
…ith a macro

Add spark_nullability!, which implements simplify, and with return_type
also return_field_from_args, from a function's SparkNullability. This
replaces the same eight-line simplify, and the three-line
return_field_from_args, in each function.

Part of apache#383.
The LIKE operator plans as DataFusion's built-in, so the like record now
calls the like function only.

Part of apache#383.
- Make SparkNullability an enum of the two nullable rules, since every
  function that uses it is null-intolerant, and drop the unused flag.
- Take &dyn ScalarUDFImpl, so the helpers are compiled once.
- Remove the NULL-folding prologue from add_months, date_diff and
  unix_date: their rewrites already fold a NULL argument.
- Share list_element_field from functions_nested_utils and use it in
  array_contains.
- Move unix_seconds, unix_millis and unix_micros onto SparkNullability.

Part of apache#383.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] make_dt_interval declares a non-nullable result but can return NULL

1 participant