Skip to content

feat(spark): implement Spark-compatible timestamp_micros - #381

Merged
andygrove merged 2 commits into
apache:mainfrom
Ritesh17rb:implement-timestamp-micros
Nov 20, 2026
Merged

andygrove merged 2 commits into
apache:mainfrom
Ritesh17rb:implement-timestamp-micros

Conversation

@Ritesh17rb

@Ritesh17rb Ritesh17rb commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #236.

Rationale for this change

Spark's timestamp_micros interprets integral input as microseconds since the UTC epoch and rejects strings, decimals, and floating-point values. DataFusion's to_timestamp_micros accepts those additional types.

What changes are included in this PR?

  • Implement and register timestamp_micros and its Rust expression helper.
  • Add shared check-only input validation for Spark's ExpectsInputTypes, including Spark-style argument errors.
  • Widen signed integers and Arrow UInt8/UInt16/UInt32 to Int64. Reject UInt64 because its full range cannot fit in the timestamp count.
  • Preserve NULLs, input nullability, the full Int64 range, and the session time zone (UTC by default).
  • Add SQL and unit coverage and generate the function reference.

Are these changes tested?

Formatting, workspace Clippy, 585 library tests, 5 documentation tests, all 516 SQL logic-test runs, and documentation generation pass. SQL tests cover integer widths, unsigned boundaries, dictionaries, NULLs, invalid types, ANSI mode, Int64 round trips, time zones, and a DST transition in both execution modes.

The repository PySpark runner passes all 20 comparable records against Spark 4.2.0. Four Arrow-specific records are skipped because Spark has no unsigned types or arrow_typeof. Detailed logs and Windows reproduction instructions are preserved in this comment.

Are there any user-facing changes?

Spark mode now provides timestamp_micros(microseconds). For example, timestamp_micros(0::BIGINT) returns the UTC epoch.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, it's a really thorough PR. I ran the PySpark runner against Spark 4.2.0 locally and all 20 records match, and the .slt passes in both modes. I especially like the i64::MIN/i64::MAX round trips and the DST transition test.

Could you trim the PR description a bit before merge? It becomes the commit message, so the Windows wrapper and MSVC details would fit better in a comment.

&self.signature
}

fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree SparkInputTypes isn't a fit as it stands, since it models ImplicitCastInputTypes and would cast '1'. timestamp_millis and timestamp_seconds will need the same ExpectsInputTypes treatment soon, though. What do you think about adding a check-only mode to implicit_cast.rs so all three share it? That would also give this error the same Spark-shaped message the other functions use ("The first parameter of timestamp_micros requires the "INTEGRAL" type..."). If you'd rather keep this PR focused, a tracking issue linked here would work too.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented in 1526fc0. Added SparkInputTypes::check_only in implicit_cast.rs and switched timestamp_micros to use it. It validates the declared input types without implicit casts, handles NULLs and dictionary layouts, and shares the Spark-style argument errors. The helper is available for timestamp_millis and timestamp_seconds when those are implemented. Added coverage for validation in both ANSI modes and the error message. Workspace Clippy and all 516 SQL test runs pass.

};
// Spark's ExpectsInputTypes accepts integers and NULL, without the
// numeric/string casts allowed by ImplicitCastInputTypes.
if value_type.is_signed_integer() || matches!(value_type, DataType::Null) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Integral rule in implicit_cast.rs uses is_integer(), so it also accepts UInt8/UInt16/UInt32. Was rejecting unsigned types here intentional? Spark never produces them, but a DataFusion caller with a UInt32 column would see different behavior here than from the other integral-argument functions. Widening those three to Int64 would be lossless.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The signed-only check was intentional to follow Spark's native input types, but I agree that accepting the safely widenable Arrow unsigned types is more consistent here. Updated in 1526fc0 to accept UInt8, UInt16, and UInt32 and widen them losslessly to Int64. UInt64 remains rejected because its full range cannot fit in the signed timestamp count; this is documented. Added SQL coverage for each accepted unsigned type's maximum value, dictionary input with NULLs, and UInt64 rejection. All 516 SQL test runs pass, including both modes for timestamp_micros.

@Ritesh17rb

Copy link
Copy Markdown
Contributor Author

Moving the detailed validation output and Windows reproduction instructions out of the PR description, as requested. These logs describe the original commit 6ebf1e3; validation of the review fixes will be summarized separately.

Captured Rust test and lint output
test function::datetime::timestamp_micros::tests::return_field_tracks_session_timezone_and_input_nullability ... ok
test result: ok. 584 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.78s
test result: ok. 5 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 0.10s

PASS [simplify] datetime\timestamp_micros.slt
PASS [evaluate] datetime\timestamp_micros.slt
516 of 516 test runs passed

    Finished `dev` profile [unoptimized] target(s) in 15.74s

dev/update_function_docs.sh: Wrote docs/functions

Formatting produced no output and exited successfully. The two ignored documentation examples are existing examples in implicit_cast.rs and lib.rs. Local builds used one build job, disabled debug symbols, and disabled incremental compilation to reduce memory use.

Captured Spark 4.2.0 comparison output
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:14: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:21: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:27: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:33: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:42: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:50: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:57: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:66: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:69: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:72: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:75: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:78: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:82: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:85: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:88: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:94: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:98: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:101: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:107: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:114: pass
crates\sqllogictest\test_files\datetime\timestamp_micros.slt:118: skipped: Spark has no function `arrow_typeof`
Spark 4.2.0: 21 records in 1 file(s)
      20  pass
       1  skipped
               1  Spark has no function `arrow_typeof`

The skipped record calls arrow_typeof, which Spark does not provide. The Arrow return type is asserted by the local SQL test and unit test above.

Spark comparison reproduction and Windows compatibility details

Install the repository's pinned PySpark dependency, then run its comparator against the changed file:

python -m pip install -r crates/sqllogictest/scripts/requirements.txt
python crates/sqllogictest/scripts/pyspark_slt_runner.py crates/sqllogictest/test_files/datetime/timestamp_micros.slt --verbose

For the local Windows verification, install tzdata for IANA time zones and save the following temporary wrapper as windows_spark_check.py in the repository root. Windows Python's datetime.fromtimestamp rejects negative epoch seconds; the wrapper decodes PySpark's returned microsecond count with UTC epoch arithmetic, as the repository comparator expects. This affects Python result conversion; Spark evaluates the original queries. The wrapper is included here for reproducibility and is not part of the library change.

"""Run the upstream Spark comparator with Windows timestamp conversion support."""
import datetime
import runpy
import sys
from pathlib import Path

from pyspark.sql.types import TimestampType


def utc_from_internal(self, timestamp):
    # The upstream comparator expects UTC datetimes. Windows' fromtimestamp
    # rejects negative epoch seconds, so use epoch arithmetic in this process.
    if timestamp is None:
        return None
    return datetime.datetime(1970, 1, 1) + datetime.timedelta(microseconds=timestamp)


TimestampType.fromInternal = utc_from_internal
runner = Path.cwd() / "crates/sqllogictest/scripts/pyspark_slt_runner.py"
sys.argv[0] = str(runner)
runpy.run_path(str(runner), run_name="__main__")

The command used locally was equivalent to:

python -m pip install tzdata
python windows_spark_check.py crates/sqllogictest/test_files/datetime/timestamp_micros.slt --verbose

On Windows, the documentation generation script was run from Git Bash with the MSVC binary directory placed first in PATH so that Rust selects Microsoft's link.exe.

@andygrove

Copy link
Copy Markdown
Member

Thanks for moving the validation logs into a comment. It looks like the PR description itself still has the Windows wrapper and MSVC details, though. Could you edit the description down to the summary sections? Also, whenever you get a chance, could you reply on the two inline threads about the shared check-only mode and the unsigned integer types?

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround. check_only fits nicely into implicit_cast.rs, and it's good to have it ready for timestamp_millis and timestamp_seconds. The unsigned widening and the UInt64 rejection make sense, and the tests cover the limits of each type. I ran the unit tests, the .slt in both modes and clippy locally, and they all pass. I've approved the CI workflows so they can run.

@andygrove
andygrove merged commit 1181abf into apache:main Nov 20, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement Spark-compatible timestamp_micros

2 participants