You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
arrays_overlap panics on nested double arrays built from expressions #5269
arrays_overlap over array<array<double>> panics in native code when both sides of the comparison are expression-constructed nested lists rather than plain columns.
org.apache.comet.CometNativeException: native panic: index out of bounds: the len is 2 but the index is 2
Comet native panic: panicked at arrow-array-58.4.0/src/array/list_array.rs:352:19
list_array.rs:352 is GenericListArray::value, so something in arrays_overlap_list_generic (native/spark-expr/src/array_funcs/arrays_overlap.rs) indexes past the end of one of its two list arrays.
Steps to reproduce
As a Comet SQL file test:
statement
CREATETABLEt_panic(a array<array<double>>) USING parquet
statement
INSERT INTO t_panic VALUES
(array(array(-0.0))),
(array(array(0.0))),
(array(array(1.0, -0.0))),
(array(array(double('NaN')))),
(array(array(1.0))),
(array(array(-0.0), cast(NULLas array<double>))),
(array(cast(NULLas array<double>)))
query
SELECT arrays_overlap(array(array(a[0][0])), array(array(cast(0.0as double)))) FROM t_panic
Reproduced on apache/main at f262b13d2, Spark 4.1 profile, JDK 17, macOS.
What I could and could not narrow down
The panic needs all seven rows. None of these reproduce it on their own:
any single row from the table, including the two rows carrying a NULL inner list
a two-row table combining a multi-element inner list with a NULL inner list
the int equivalent of the same query
arrays_overlap(a, array(array(cast(0.0 as double)))), a plain column against a constant
arrays_overlap(array(a[0]), array(array(0))), only the outer list constructed
The failing task index moves with the number of rows, so this looks like a batch or partition boundary interacting with the per-row left.value(i) / right.value(i) slicing in arrays_overlap_list_generic, rather than anything about a specific row's contents. Worth checking whether the two sides can end up with different len() when one of them arrives as a broadcast scalar and the other is a per-row constructed array.
Additional context
Found while reviewing #5235, which changes the same function. The panic reproduces identically with and without that PR applied, so it is independent of it.
Activity
peterxcli commentedon Aug 19, 2026
take