Repository navigation
feat(spark): implement Spark-compatible timestamp_millis - #392
Ritesh17rb wants to merge 1 commit into
Conversation
|
Detailed validation for commit The main description is intentionally limited to the summary sections. Captured output, source audit notes, and platform-specific reproduction details are below. Captured test, lint, and generation resultsThe new SQL file was run against the unfixed base before adding the implementation: After implementing the function: Full library and documentation tests: The two ignored documentation examples already exist in Full SQL suite: Spark comparison of the changed file: All four Arrow-specific records pass locally in both SQL modes. Additional live Spark BIGINT probes confirm the numeric results used for the unsigned-value cases. A separate Spark 4.2.0 ground-truth harness confirmed exact Formatting exited successfully with no differences. Workspace Clippy with all targets/features and Regeneration also includes Validation commands and native Windows reproductionFrom the repository root: cargo fmt --all -- --check
cargo test --locked -p datafusion-spark --lib timestamp_millis
cargo test --locked -p datafusion-spark-sqllogictest -- timestamp_millis
cargo test --locked -p datafusion-spark
cargo test --locked -p datafusion-spark-sqllogictest
cargo clippy --locked --all-targets --all-features --workspace -- -D warnings
dev/update_function_docs.shUsing the basename as the SQL test filter works with both Windows and Unix path separators. No expected results were generated with The repository's Spark comparator can be run with its pinned dependency: python -m pip install -r crates/sqllogictest/scripts/requirements.txt
python crates/sqllogictest/scripts/pyspark_slt_runner.py crates/sqllogictest/test_files/datetime/timestamp_millis.slt --verboseLocal environment: native Windows 11 x64, Rust stable 1.100.0/MSVC, Python 3.12, PySpark 4.2.0, and JDK 20.0.2. Builds used one job, On Windows, the comparator used the same UTC epoch-arithmetic result converter documented in the previous PR's validation comment, because Windows Python rejects negative epoch seconds in The documentation script was run from Git Bash with the configured MSVC binary directory first in PATH. The local wrapper initializes the Visual Studio environment once. Cargo 1.100 prints pre-existing workspace-manifest warnings for Spark source audit and coverageFunction summary: integral milliseconds since the UTC epoch become a timestamp with a signed 64-bit microsecond count. NULL remains NULL. The input's nullability is preserved, and the session time zone affects display. Version comparison: read the registries, Reference: Spark 4.2.0 conversion implementation and its unit tests. Implementation review:
Live Spark additionally confirmed that setting Findings: the function was absent and the SQL file was a commented-out stub. This PR implements it and activates the regression coverage. No deferred divergence or additional coverage gap was identified within this function's scope. Comet and Sail were consulted as peer references; Spark source and live Spark results set the expectations. Surfaced by the audit-datafusion-spark-expression skill. |
Which issue does this PR close?
Closes #235.
Rationale for this change
Spark's
timestamp_millisaccepts integral input and converts milliseconds to microseconds with checked multiplication. DataFusion'sto_timestamp_millisaccepts additional types and stores milliseconds, giving it a different range and overflow behavior.What changes are included in this PR?
timestamp_milliswith scalar and array execution, microsecond precision, session-time-zone metadata, and input nullability.SparkInputTypes::check_onlyfrom feat(spark): implement Spark-compatible timestamp_micros #381 and its lossless integer-widening policy. Accept Arrow UInt8/UInt16/UInt32 and reject UInt64.long overflowoutside the valid millisecond range in both ANSI modes. Add SQL/unit coverage and regenerate the function reference.Are these changes tested?
Formatting, workspace Clippy with all targets/features, 594 unit tests, 3 documentation tests, all 516 SQL test runs, and documentation generation pass. Two existing documentation examples are ignored.
The changed SQL file passes in both execution modes. The PySpark runner matches all 38 comparable records against Spark 4.2.0; four Arrow-specific records are checked locally. Coverage includes both overflow boundaries, negative values, NULLs, integer widths, dictionaries, rejected types, time zones, and DST.
Detailed validation output and reproduction instructions are in this comment.
Are there any user-facing changes?
Spark mode now provides
timestamp_millis(milliseconds). For example,timestamp_millis(-1::BIGINT)returns one millisecond before the UTC epoch.Surfaced by the audit-datafusion-spark-expression skill.