Skip to content

feat: add spark-compatible is_variant_null - #394

Open
Pastoray wants to merge 1 commit into
apache:mainfrom
Pastoray:feat/is-variant-null
Open

Pastoray wants to merge 1 commit into
apache:mainfrom
Pastoray:feat/is-variant-null

Conversation

@Pastoray

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

is_variant_null is a Spark 4.0+ function that checks whether a VARIANT holds a
JSON null. It is not implemented in this crate, so a query that calls it fails
with an unknown function error.

What changes are included in this PR?

  • Adds is_variant_null(v) in a new function/variant module, registered in
    all_default_scalar_functions and expr_fn.
  • Returns true only when the first byte of value is 0, which is a variant
    null. Every other value is false, including a SQL NULL, so the result is a
    non-nullable BOOLEAN.
  • An empty value raises [MALFORMED_VARIANT] (SQLSTATE 22023), with the same
    message as Spark.
  • Only a VARIANT is accepted, with no implicit casts, as in Spark. A bare NULL
    literal is accepted and returns false, as in Spark.
  • This crate has no VARIANT type or parse_json yet, so a VARIANT is read as the
    struct of the arrow.parquet.variant extension type: binary metadata and
    value fields, as Binary, LargeBinary or BinaryView. Extra shredding fields
    such as typed_value are ignored. The function can adopt a real VARIANT type
    once Arrow and DataFusion have one.
  • Regenerates docs/functions with a new "Variant Functions" page.

Are these changes tested?

Yes.

  • Unit tests in is_variant_null.rs cover the header byte, SQL NULL rows, a NULL
    value, the malformed case, the three binary layouts, field order, shredded
    structs, scalar input, the non-nullable return field and the accepted
    argument types.
  • variant/is_variant_null.slt covers the same cases through SQL. It runs in both
    the simplify and evaluate modes and includes ANSI mode on and off.
  • I checked the expected values on Spark 4.2.0 with the PySpark runner
    (pyspark_slt_runner.py) and against the Spark 4.2.0 source for IsVariantNull
    and VariantExpressionEvalUtils.isVariantNull. Spark builds a VARIANT with
    parse_json, which this crate does not have, so the records that build one with
    named_struct are marked skipif spark. The runner passes the records it runs
    (10 pass, 10 skipped).
  • I also flipped the header comparison locally to confirm that both the unit
    tests and the .slt fail.

Are there any user-facing changes?

Yes, a new function: is_variant_null. There are no changes to existing
functions and no breaking changes to the public Rust API.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement Spark-compatible is_variant_null

1 participant