Repository navigation
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
is_variant_null#277.Rationale for this change
is_variant_nullis a Spark 4.0+ function that checks whether a VARIANT holds aJSON
null. It is not implemented in this crate, so a query that calls it failswith an unknown function error.
What changes are included in this PR?
is_variant_null(v)in a newfunction/variantmodule, registered inall_default_scalar_functionsandexpr_fn.trueonly when the first byte ofvalueis0, which is a variantnull. Every other value is
false, including a SQL NULL, so the result is anon-nullable BOOLEAN.
valueraises[MALFORMED_VARIANT](SQLSTATE 22023), with the samemessage as Spark.
NULLliteral is accepted and returns
false, as in Spark.parse_jsonyet, so a VARIANT is read as thestruct of the
arrow.parquet.variantextension type: binarymetadataandvaluefields, as Binary, LargeBinary or BinaryView. Extra shredding fieldssuch as
typed_valueare ignored. The function can adopt a real VARIANT typeonce Arrow and DataFusion have one.
docs/functionswith a new "Variant Functions" page.Are these changes tested?
Yes.
is_variant_null.rscover the header byte, SQL NULL rows, a NULLvalue, the malformed case, the three binary layouts, field order, shreddedstructs, scalar input, the non-nullable return field and the accepted
argument types.
variant/is_variant_null.sltcovers the same cases through SQL. It runs in boththe simplify and evaluate modes and includes ANSI mode on and off.
(
pyspark_slt_runner.py) and against the Spark 4.2.0 source forIsVariantNulland
VariantExpressionEvalUtils.isVariantNull. Spark builds a VARIANT withparse_json, which this crate does not have, so the records that build one withnamed_structare markedskipif spark. The runner passes the records it runs(10 pass, 10 skipped).
tests and the
.sltfail.Are there any user-facing changes?
Yes, a new function:
is_variant_null. There are no changes to existingfunctions and no breaking changes to the public Rust API.