Repository navigation
feat(spark): implement Spark-compatible timestamp_micros - #381
Conversation
andygrove
left a comment
There was a problem hiding this comment.
Thanks for this, it's a really thorough PR. I ran the PySpark runner against Spark 4.2.0 locally and all 20 records match, and the .slt passes in both modes. I especially like the i64::MIN/i64::MAX round trips and the DST transition test.
Could you trim the PR description a bit before merge? It becomes the commit message, so the Windows wrapper and MSVC details would fit better in a comment.
| &self.signature | ||
| } | ||
|
|
||
| fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> { |
There was a problem hiding this comment.
I agree SparkInputTypes isn't a fit as it stands, since it models ImplicitCastInputTypes and would cast '1'. timestamp_millis and timestamp_seconds will need the same ExpectsInputTypes treatment soon, though. What do you think about adding a check-only mode to implicit_cast.rs so all three share it? That would also give this error the same Spark-shaped message the other functions use ("The first parameter of timestamp_micros requires the "INTEGRAL" type..."). If you'd rather keep this PR focused, a tracking issue linked here would work too.
There was a problem hiding this comment.
Implemented in 1526fc0. Added SparkInputTypes::check_only in implicit_cast.rs and switched timestamp_micros to use it. It validates the declared input types without implicit casts, handles NULLs and dictionary layouts, and shares the Spark-style argument errors. The helper is available for timestamp_millis and timestamp_seconds when those are implemented. Added coverage for validation in both ANSI modes and the error message. Workspace Clippy and all 516 SQL test runs pass.
| }; | ||
| // Spark's ExpectsInputTypes accepts integers and NULL, without the | ||
| // numeric/string casts allowed by ImplicitCastInputTypes. | ||
| if value_type.is_signed_integer() || matches!(value_type, DataType::Null) { |
There was a problem hiding this comment.
The Integral rule in implicit_cast.rs uses is_integer(), so it also accepts UInt8/UInt16/UInt32. Was rejecting unsigned types here intentional? Spark never produces them, but a DataFusion caller with a UInt32 column would see different behavior here than from the other integral-argument functions. Widening those three to Int64 would be lossless.
There was a problem hiding this comment.
The signed-only check was intentional to follow Spark's native input types, but I agree that accepting the safely widenable Arrow unsigned types is more consistent here. Updated in 1526fc0 to accept UInt8, UInt16, and UInt32 and widen them losslessly to Int64. UInt64 remains rejected because its full range cannot fit in the signed timestamp count; this is documented. Added SQL coverage for each accepted unsigned type's maximum value, dictionary input with NULLs, and UInt64 rejection. All 516 SQL test runs pass, including both modes for timestamp_micros.
|
Moving the detailed validation output and Windows reproduction instructions out of the PR description, as requested. These logs describe the original commit Captured Rust test and lint outputFormatting produced no output and exited successfully. The two ignored documentation examples are existing examples in Captured Spark 4.2.0 comparison outputThe skipped record calls Spark comparison reproduction and Windows compatibility detailsInstall the repository's pinned PySpark dependency, then run its comparator against the changed file: python -m pip install -r crates/sqllogictest/scripts/requirements.txt
python crates/sqllogictest/scripts/pyspark_slt_runner.py crates/sqllogictest/test_files/datetime/timestamp_micros.slt --verboseFor the local Windows verification, install """Run the upstream Spark comparator with Windows timestamp conversion support."""
import datetime
import runpy
import sys
from pathlib import Path
from pyspark.sql.types import TimestampType
def utc_from_internal(self, timestamp):
# The upstream comparator expects UTC datetimes. Windows' fromtimestamp
# rejects negative epoch seconds, so use epoch arithmetic in this process.
if timestamp is None:
return None
return datetime.datetime(1970, 1, 1) + datetime.timedelta(microseconds=timestamp)
TimestampType.fromInternal = utc_from_internal
runner = Path.cwd() / "crates/sqllogictest/scripts/pyspark_slt_runner.py"
sys.argv[0] = str(runner)
runpy.run_path(str(runner), run_name="__main__")The command used locally was equivalent to: python -m pip install tzdata
python windows_spark_check.py crates/sqllogictest/test_files/datetime/timestamp_micros.slt --verboseOn Windows, the documentation generation script was run from Git Bash with the MSVC binary directory placed first in PATH so that Rust selects Microsoft's |
|
Thanks for moving the validation logs into a comment. It looks like the PR description itself still has the Windows wrapper and MSVC details, though. Could you edit the description down to the summary sections? Also, whenever you get a chance, could you reply on the two inline threads about the shared check-only mode and the unsigned integer types? |
andygrove
left a comment
There was a problem hiding this comment.
Thanks for the quick turnaround. check_only fits nicely into implicit_cast.rs, and it's good to have it ready for timestamp_millis and timestamp_seconds. The unsigned widening and the UInt64 rejection make sense, and the tests cover the limits of each type. I ran the unit tests, the .slt in both modes and clippy locally, and they all pass. I've approved the CI workflows so they can run.
Which issue does this PR close?
Closes #236.
Rationale for this change
Spark's
timestamp_microsinterprets integral input as microseconds since the UTC epoch and rejects strings, decimals, and floating-point values. DataFusion'sto_timestamp_microsaccepts those additional types.What changes are included in this PR?
timestamp_microsand its Rust expression helper.ExpectsInputTypes, including Spark-style argument errors.Are these changes tested?
Formatting, workspace Clippy, 585 library tests, 5 documentation tests, all 516 SQL logic-test runs, and documentation generation pass. SQL tests cover integer widths, unsigned boundaries, dictionaries, NULLs, invalid types, ANSI mode, Int64 round trips, time zones, and a DST transition in both execution modes.
The repository PySpark runner passes all 20 comparable records against Spark 4.2.0. Four Arrow-specific records are skipped because Spark has no unsigned types or
arrow_typeof. Detailed logs and Windows reproduction instructions are preserved in this comment.Are there any user-facing changes?
Spark mode now provides
timestamp_micros(microseconds). For example,timestamp_micros(0::BIGINT)returns the UTC epoch.