Skip to content

docs(spark): document string functions with #[user_doc] - #105

Merged
andygrove merged 2 commits into
apache:mainfrom
andygrove:docs-user-doc-string
Nov 14, 2026
Merged

andygrove merged 2 commits into
apache:mainfrom
andygrove:docs-user-doc-string

Conversation

@andygrove

@andygrove andygrove commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Part of #55

Rationale

#58 added #[user_doc] support and documented sha2 and substring as examples. This PR applies the same approach to the rest of the general-purpose string functions, so they appear in generated function references.

Changes

  • Document ascii, char, concat, concat_ws, elt, format_string (alias printf), length (aliases char_length, character_length, len), quote, space, soundex, like and ilike with #[user_doc], and implement documentation() by returning self.doc().
  • Each description covers how the function differs from the DataFusion built-in of the same name, or says there isn't one, plus its Spark edge cases: NULL handling, out-of-range values, ANSI mode for elt, and accepted input types.
  • Cut the structs' rustdoc down to a one-line summary and the Spark docs link. The behavior notes that were there for ascii, concat and concat_ws moved into the #[user_doc] text, so nothing is written twice.
  • space only accepts INT, and integer literals are BIGINT in DataFusion, so its docs say to cast and its examples use ::INT.
  • The rest of the string module will come in separate PRs. The binary, encoding and UTF-8 functions (to_binary, try_to_binary, encode, base64, unbase64, make_valid_utf8, is_valid_utf8, luhn_check) go in their own PR. substring was done in docs: document sha2 and substring with #[user_doc] #58, and regexp_extract is documented in its own PR (feat(spark): add Spark-compatible regexp_extract function #73).

Testing

  • cargo fmt --all, cargo clippy --all-targets --all-features -- -D warnings, cargo test -p datafusion-spark and cargo test -p datafusion-spark-sqllogictest pass.
  • I ran every SQL example in the docs through the sqllogictest runner, and they all match.
  • The documented behavior is covered by the existing .slt and unit tests for these functions, or is visible in the code (for example, like and ilike take exactly two arguments, so Spark's escape argument isn't supported).

User-facing changes

documentation() now returns docs for these 12 functions. Behavior does not change.

Known gaps

The docs describe these differences from Spark, link them to their issues, and new sqllogictest cases cover the current behavior:

🤖 Generated with Claude Code

Document ascii, char, concat, concat_ws, elt, format_string, length,
quote, space, soundex, like and ilike with #[user_doc] and implement
documentation(), following the guidelines in CONTRIBUTING.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add a sqllogictest case showing that space(2) fails to plan because the
literal is BIGINT.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@coderfender coderfender left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@andygrove
andygrove merged commit 9fad0d1 into apache:main Nov 14, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants