Skip to content

feat(spark): implement Spark-compatible find_in_set - #373

Open
umrndem wants to merge 2 commits into
apache:mainfrom
umrndem:fix/spark-find-in-set
Open

umrndem wants to merge 2 commits into
apache:mainfrom
umrndem:fix/spark-find-in-set

Conversation

@umrndem

@umrndem umrndem commented Nov 19, 2026

Copy link
Copy Markdown

Which issue does this PR close?

Closes #200.

What changes are included in this PR?

Register find_in_set with Spark’s string coercion rules and reuse DataFusion’s existing search implementation. The result is always INT, including for LargeUtf8 inputs.

Are these changes tested?

Added SQL cases for nulls, implicit casts, mixed string layouts, and the return type. All 562 Rust tests and 258 SQL test files passed. Formatting and Clippy passed. I also checked 20 cases against Spark 4.2.0’s string implementation.

Are there any user-facing changes?

find_in_set now uses Spark-compatible behavior for the default binary collation.

Register find_in_set with Spark string coercion and an INT result for every string layout. Reuse the core search kernel and add SQL coverage for nulls, casts, mixed string layouts, and the return type.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. Reusing the core kernel and only fixing the layout and return type keeps it small, and the mixed-layout and arrow_typeof tests pin exactly the parts that differ from core. I rebased it onto main locally (to pick up #316) and the file passes in both the simplify and evaluate modes. The PySpark runner agrees on every record except the marked collation one.

Int32
Int32

# xfail spark: DataFusion SQL does not support explicit collations.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this marker link the collation tracking issue, so it gets cleaned up when collations land? Something like # xfail spark: collations are not supported yet (https://github.com/apache/datafusion-spark/issues/299).


# Spark implicitly casts both arguments to strings.
query II
SELECT find_in_set(2::INT, '1,2,3'), find_in_set('2', 2::INT);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice coverage of the int and boolean casts. Binary and date go through different branches of implicit_cast, so it might be worth pinning one of each too. find_in_set(X'62', 'a,b') and find_in_set(DATE '2020-01-01', 'x,2020-01-01') both return 2 here and on Spark 4.2.0.

@umrndem

umrndem commented Nov 19, 2026

Copy link
Copy Markdown
Author

thanks, i linked #299 in the collation comment and added the binary and date cases. the find_in_set tests pass in both simplify and evaluate modes.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround on both points. Since this branched, #115 landed on main and CI now checks that docs/functions matches the #[user_doc] attributes. Could you rebase onto main and run dev/update_function_docs.sh? It adds find_in_set to docs/functions/README.md and docs/functions/scalar/string.md, and the lint job will fail without it.

2

query I
SELECT find_in_set(DATE '2020-01-01', 'x,2020-01-01');

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The DATE case matches Spark, but a few other implicit casts don't yet, because the planner uses DataFusion's cast rather than Spark's. On Spark 4.2.0, find_in_set(TIMESTAMP '2020-01-01 00:00:00', 'x,2020-01-01 00:00:00') returns 2 and find_in_set(1e20::DOUBLE, '1.0E20,100000000000000000000') returns 1, while here both return 0. find_in_set(X'FF', 'a,b') returns 0 on Spark but fails here with Encountered non UTF-8 data. All three are covered by #88. Could you add them as records with the current DataFusion results (and statement error for the binary one), each with an # xfail spark: marker that links #88? That way they get flipped when the cast port lands instead of being forgotten.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement Spark-compatible find_in_set

2 participants