Skip to content

feat(spark): implement Spark-compatible timestamp_millis - #392

Open
Ritesh17rb wants to merge 1 commit into
apache:mainfrom
Ritesh17rb:implement-timestamp-millis
Open

Ritesh17rb wants to merge 1 commit into
apache:mainfrom
Ritesh17rb:implement-timestamp-millis

Conversation

@Ritesh17rb

@Ritesh17rb Ritesh17rb commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #235.

Rationale for this change

Spark's timestamp_millis accepts integral input and converts milliseconds to microseconds with checked multiplication. DataFusion's to_timestamp_millis accepts additional types and stores milliseconds, giving it a different range and overflow behavior.

What changes are included in this PR?

  • Implement and register timestamp_millis with scalar and array execution, microsecond precision, session-time-zone metadata, and input nullability.
  • Reuse SparkInputTypes::check_only from feat(spark): implement Spark-compatible timestamp_micros #381 and its lossless integer-widening policy. Accept Arrow UInt8/UInt16/UInt32 and reject UInt64.
  • Raise long overflow outside the valid millisecond range in both ANSI modes. Add SQL/unit coverage and regenerate the function reference.

Are these changes tested?

Formatting, workspace Clippy with all targets/features, 594 unit tests, 3 documentation tests, all 516 SQL test runs, and documentation generation pass. Two existing documentation examples are ignored.

The changed SQL file passes in both execution modes. The PySpark runner matches all 38 comparable records against Spark 4.2.0; four Arrow-specific records are checked locally. Coverage includes both overflow boundaries, negative values, NULLs, integer widths, dictionaries, rejected types, time zones, and DST.

Detailed validation output and reproduction instructions are in this comment.

Are there any user-facing changes?

Spark mode now provides timestamp_millis(milliseconds). For example, timestamp_millis(-1::BIGINT) returns one millisecond before the UTC epoch.

Surfaced by the audit-datafusion-spark-expression skill.

@Ritesh17rb

Copy link
Copy Markdown
Contributor Author

Detailed validation for commit 18a4a00.

The main description is intentionally limited to the summary sections. Captured output, source audit notes, and platform-specific reproduction details are below.

Captured test, lint, and generation results

The new SQL file was run against the unfixed base before adding the implementation:

FAIL [simplify] datetime\timestamp_millis.slt
Error during planning: Invalid function 'timestamp_millis'.
FAIL [evaluate] datetime\timestamp_millis.slt
Error during planning: Invalid function 'timestamp_millis'.
0 of 2 test runs passed

After implementing the function:

test function::datetime::timestamp_millis::tests::return_field_tracks_session_timezone_and_input_nullability ... ok
test function::datetime::timestamp_millis::tests::null_slots_do_not_overflow ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 592 filtered out

PASS [simplify] datetime\timestamp_millis.slt
PASS [evaluate] datetime\timestamp_millis.slt
2 of 2 test runs passed

Full library and documentation tests:

test result: ok. 594 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

test result: ok. 3 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out

The two ignored documentation examples already exist in implicit_cast.rs and lib.rs.

Full SQL suite:

516 of 516 test runs passed

Spark comparison of the changed file:

Spark 4.2.0: 42 records in 1 file(s)
      38  pass
       4  skipped
               3  arrow_cast to an unsigned integer, which Spark does not have
               1  Spark has no function `arrow_typeof`

All four Arrow-specific records pass locally in both SQL modes. Additional live Spark BIGINT probes confirm the numeric results used for the unsigned-value cases. A separate Spark 4.2.0 ground-truth harness confirmed exact long overflow errors in both ANSI modes, including scalar and column inputs.

Formatting exited successfully with no differences. Workspace Clippy with all targets/features and -D warnings exited with code 0. The documentation generator completed with:

Wrote docs/functions
DOCS_EXIT_CODE=0

Regeneration also includes dayofweek's existing #[user_doc] content, whose generated entry was missing from base 1181abf.

Validation commands and native Windows reproduction

From the repository root:

cargo fmt --all -- --check
cargo test --locked -p datafusion-spark --lib timestamp_millis
cargo test --locked -p datafusion-spark-sqllogictest -- timestamp_millis
cargo test --locked -p datafusion-spark
cargo test --locked -p datafusion-spark-sqllogictest
cargo clippy --locked --all-targets --all-features --workspace -- -D warnings
dev/update_function_docs.sh

Using the basename as the SQL test filter works with both Windows and Unix path separators. No expected results were generated with --complete.

The repository's Spark comparator can be run with its pinned dependency:

python -m pip install -r crates/sqllogictest/scripts/requirements.txt
python crates/sqllogictest/scripts/pyspark_slt_runner.py crates/sqllogictest/test_files/datetime/timestamp_millis.slt --verbose

Local environment: native Windows 11 x64, Rust stable 1.100.0/MSVC, Python 3.12, PySpark 4.2.0, and JDK 20.0.2. Builds used one job, CARGO_PROFILE_DEV_DEBUG=0, and CARGO_INCREMENTAL=0 to limit memory use. The DataFusion dependency resolved to branch-56 commit 3f197fe0.

On Windows, the comparator used the same UTC epoch-arithmetic result converter documented in the previous PR's validation comment, because Windows Python rejects negative epoch seconds in datetime.fromtimestamp. Spark evaluated the original queries. The separate ground-truth harness collected timestamp strings and integer counts to avoid that Python conversion limitation.

The documentation script was run from Git Bash with the configured MSVC binary directory first in PATH. The local wrapper initializes the Visual Studio environment once. Cargo 1.100 prints pre-existing workspace-manifest warnings for readme and rust-version; these also appear in the baseline run. Rust Clippy itself passed with warnings denied.

Spark source audit and coverage

Function summary: integral milliseconds since the UTC epoch become a timestamp with a signed 64-bit microsecond count. NULL remains NULL. The input's nullability is preserved, and the session time zone affects display.

Version comparison: read the registries, IntegralToTimestampBase, MillisToTimestamp, relevant unit tests, and timestamp golden outputs for Spark 3.5.8, 4.0.4, 4.1.3, and 4.2.0. No relevant behavior difference was found. The conversion uses Math.multiplyExact(value, 1000) in both interpreted and generated execution. Changing ANSI's default in Spark 4.x does not affect this function.

Reference: Spark 4.2.0 conversion implementation and its unit tests.

Implementation review: SparkInputTypes::check_only preserves the strict integral signature and shared argument errors from #381. Accepted widths widen to Int64 before the checked kernel. Scalar and array paths share checked_mul(1000). The result carries microsecond precision and session-time-zone metadata. Comet's physical planner was also checked: it calls coerce_types() and inserts the returned widening casts before invoking the physical expression.

Dimension Coverage
Scalar and column execution SQL, in both execution modes
Zero, negatives, pre-epoch timestamps SQL and live Spark
Untyped/typed NULL, mixed and all-null columns SQL and live Spark
Overflowing storage beneath an Arrow null mask Direct Rust invocation test
Both inclusive range endpoints Integer round trips, avoiding calendar-format limits
First out-of-range values and Int64 extrema Scalar/column errors, ANSI on and off
Signed integer widths SQL and live Spark
FLOAT/DOUBLE/DECIMAL/STRING/BOOLEAN rejection Both ANSI modes
Dictionaries and Arrow unsigned types Local SQL in both modes
Time zones and DST Los Angeles display plus unchanged epoch counts
Return precision, time zone, and nullability Arrow SQL assertion and Rust metadata matrix
Floating-point special values, collation, containers Not applicable: non-integral inputs are rejected

Live Spark additionally confirmed that setting spark.sql.timestampType=TIMESTAMP_NTZ leaves this function's result as timestamp.

Findings: the function was absent and the SQL file was a commented-out stub. This PR implements it and activates the regression coverage. No deferred divergence or additional coverage gap was identified within this function's scope. Comet and Sail were consulted as peer references; Spark source and live Spark results set the expectations.

Surfaced by the audit-datafusion-spark-expression skill.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement Spark-compatible timestamp_millis

1 participant