You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native regexp_extract and regexp_extract_all error on an out-of-range idx even when no row matches #7685
The native regexp_extract and regexp_extract_all validate idx against the pattern's group count before they look at any row. Spark validates it only for a row where the pattern matches.
RegExpExtract.nullSafeEval calls checkGroupIndex inside if (m.find). A row with no match returns ''.
RegExpExtractAll.nullSafeEval calls checkGroupIndex inside while (m.find). A row with no match returns an empty array.
In Comet, parse_args in native/spark-expr/src/string_funcs/regexp_extract_common.rs computes group_count and returns an error for idx < 0 || idx > group_count before any row is evaluated. Both functions share it.
So a query where Spark succeeds fails in Comet. For example, regexp_extract(s, '\d+') uses the default idx of 1 and the pattern has no groups. Spark returns '' for every row that does not match, and errors only if some row matches. Comet errors on the whole query, even when no row matches.
The native path is reached only with spark.comet.expression.RegExpExtract.allowIncompatible=true (or RegExpExtractAll) and a literal pattern and idx. The default codegen dispatcher path runs Spark's own code and is not affected.
Found while reading the review of the same bug in the datafusion-spark port of this function (apache/datafusion-spark#73, since closed). I confirmed the Spark behavior in the 4.2.0 source. I have not run the repro below against Comet. The Comet side comes from reading the code.
Steps to reproduce
-- spark.comet.expression.RegExpExtract.allowIncompatible=trueCREATETABLEt(s string) USING parquet;
INSERT INTO t VALUES ('abc'), ('def');
SELECT regexp_extract(s, '\\d+') FROM t;
-- Spark: '', ''-- Comet: error, "Expects group index between 0 and 0, but got 1."
The same applies to regexp_extract_all with an out-of-range idx and no matching row.
Expected behavior
Validate idx only when a row matches, as Spark does. In extract_array and extract_one, check group_idx only after captures_read or captures returns a match. Return the empty result otherwise.
Add tests in regexp_extract.rs and regexp_extract_all.rs for:
no match with an out-of-range idx, which returns the empty result
a match with an out-of-range idx, which still errors
a column where only some rows match
Additional context
A related difference is that an invalid literal pattern is compiled eagerly, so a query over an all-NULL subject column errors in Comet. Spark evaluates a NULL subject row without compiling the pattern and returns NULL. Fixing that means compiling lazily on the first non-NULL row. It could go in the same change or be tracked separately.
The doc comment on spark_regexp_extract says "Errors when idx is out of range" and does not mention the match condition. It should be updated with the fix.
Activity