Skip to content

feat(spark): implement Spark-compatible log1p - #344

Open
kazantsev-maksim wants to merge 2 commits into
apache:mainfrom
kazantsev-maksim:log1p
Open

kazantsev-maksim wants to merge 2 commits into
apache:mainfrom
kazantsev-maksim:log1p

Conversation

@kazantsev-maksim

@kazantsev-maksim kazantsev-maksim commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Spark SQL provides log1p, which computes the natural logarithm of 1 + x accurately for values near zero. Adding this function to DataFusion Spark lets users run queries that use log1p with Spark-compatible behavior.

What changes are included in this PR?

  • Add and register the log1p scalar function.
  • Return NULL for inputs less than or equal to -1.0, including negative infinity.
  • Preserve NULL inputs and handle NaN, positive infinity, and values near zero.
  • Add function documentation and SQL logic tests for ordinary values, boundary cases, special values, small inputs, and supported input types.

Are these changes tested?

SQL logic tests have been added in math/log1p.slt. They have not yet been run, and the expected results have not been independently verified against Spark.

Are there any user-facing changes?

Yes. This PR adds the new log1p(expr) SQL function. It returns a DOUBLE and returns NULL for inputs less than or equal to -1.0. No existing functions are changed.

Kazantsev Maksim added 2 commits November 17, 2026 21:07
# Conflicts:
#	crates/spark/src/function/math/mod.rs
@kazantsev-maksim kazantsev-maksim changed the title impl log1p feat(spark): implement Spark-compatible log1p Nov 17, 2026

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking up log1p. The <= -1 → NULL handling matches Spark's UnaryLogExpression, and I checked the values in this file against PySpark 4.2.0 with ANSI both on and off. They all agree.

When I ran cargo test -p datafusion-spark-sqllogictest -- math/log1p locally it failed, though. Could you rebase on main and run it? There are three issues:

  • The slt runner prints doubles with 15 significant digits, so 0.4054651081081644 needs to be 0.405465108108164. The same goes for the 0.693… values.
  • The two equality queries return a boolean, so they need query B rather than query T.
  • SELECT log1p('0.5') fails at planning time. See the comment on the signature.

The PySpark runner in crates/sqllogictest/scripts is a quick way to check the file against Spark once it passes.

impl SparkLog1p {
pub fn new() -> Self {
Self {
signature: Signature::exact(vec![DataType::Float64], Volatility::Immutable),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spark's Log1p declares inputTypes = Seq(DoubleType) with ImplicitCastInputTypes. That means it accepts strings (and errors on a malformed one under ANSI), and it rejects BOOLEAN, DATE and BINARY with UNEXPECTED_INPUT_TYPE. Signature::exact(Float64) rejects strings, so log1p('0.5') in this PR's own test fails.

Since #350 we have SparkInputTypes for exactly this case. Could you switch to SparkInputTypes::new(vec![SparkInputType::Double]) and add coerce_types and with_updated_config the way hash/sha2.rs on main does? Then the argument is always Float64, and the two unsupported_data_type_exec_err arms can go.

query R
SELECT log1p('0.5');
----
0.4054651081081644

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add a few cases for the inputs Spark rejects, like log1p(true), log1p(DATE '2020-01-01') and log1p(X'01'), as query error? A malformed string like log1p('abc') would also be useful. It returns NULL with ANSI off and raises CAST_INVALID_INPUT with ANSI on, so it needs the ANSI wrapper described in test_files/README.md.

@andygrove

Copy link
Copy Markdown
Member

A correction to my earlier comment. Once this is rebased on main the 15 significant digit point no longer applies. #333 switched the slt runner to print doubles at full precision, so values like 0.4054651081081644 are already correct as written and should stay as they are. I merged main into the branch locally and those queries pass. The remaining failures were the query T to query B change and the log1p('0.5') planning error.

Also, since #115 landed, CI checks that docs/functions matches the #[user_doc] attributes. After rebasing, could you run dev/update_function_docs.sh and commit the changes it makes under docs/functions? For this PR that is docs/functions/README.md and docs/functions/scalar/math.md.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement Spark-compatible log1p

2 participants