Repository navigation
fix: [branch-1.2] fall back to Spark for a nondeterministic child of size, arrays_zip and map_from_arrays - #7670
Draft
dwsmith1983 wants to merge 2 commits into
Conversation
…size, arrays_zip and map_from_arrays
…erministic-fallback-branch-1.2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Part of #5781 on
branch-1.2. #5867 fixes it onmainand closes the issue; it is approved but not merged yet, so this should merge after it.Rationale for this change
size,arrays_zipandmap_from_arraysevaluate a nondeterministic child twice in 1.2.0: once as the value and again inside the nativeNULLguard. A stateful child such asIF(monotonically_increasing_id() % 2 = 0, array(1), NULL)then returns wrong results.#5867 routes these cases through the JVM codegen dispatcher, but it also moves map lookups with float, collated and complex keys to the dispatcher (#5580), an enhancement that #5875 measured at 9 to 11 percent slower for lookup-only DOUBLE-key projections. That does not belong in a patch release, so this change falls back to Spark instead, as
branch-1.2already does forarray_appendand ANSIelement_at.What changes are included in this PR?
sizeandarrays_zipreport a nondeterministic child as unsupported, which falls back to Spark.map_from_arraysdeclines a nondeterministic child when it is converted, since reporting it unsupported would send it to the codegen dispatcher. This also covers theallowIncompatibleroute for float keys.map_from_arraysfallback, because the generator would describe that function's unsupported reasons as running through the codegen dispatcher.ArraysZipand a newSizesection. The pages for Spark 3.5, 4.0, 4.1 and 4.2 were regenerated; the Spark 3.4 page got the same lines by hand.A non-nullable nondeterministic child, such as
size(array(monotonically_increasing_id())), also falls back now. 1.2.0 ran it natively and correctly, since the guard passes every row.How are these changes tested?
Three SQL file tests, adapted from #5867, check each function against Spark: a nondeterministic nullable child falls back, a non-nullable one falls back, and a deterministic or NULL input still runs natively. For
map_from_arrays, DOUBLE keys go to the dispatcher by default and fall back withallowIncompatible=true. Onbranch-1.2without the fix, all four nondeterministic cases and theallowIncompatiblecase return wrong results. With it, the tests pass on Spark 3.5, 4.0, 4.1 and 4.2, and the existingsize,arrays_zipandmap_from_arraystests and the array and map expression suites still pass on 4.1.