Repository navigation
feat(spark): implement Spark-compatible timestamp_micros #381
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,161 @@ | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
| // SPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership | ||
| // SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements | ||
|
|
||
| use std::sync::Arc; | ||
|
|
||
| use arrow::datatypes::{DataType, Field, FieldRef, TimeUnit}; | ||
| use datafusion_common::config::ConfigOptions; | ||
| use datafusion_common::utils::take_function_args; | ||
| use datafusion_common::{Result, internal_err, plan_err}; | ||
| use datafusion_expr::{ | ||
| ColumnarValue, Documentation, ReturnFieldArgs, ScalarFunctionArgs, ScalarUDF, | ||
| ScalarUDFImpl, Signature, Volatility, | ||
| }; | ||
| use datafusion_macros::user_doc; | ||
|
|
||
| use crate::function::implicit_cast::{SparkInputType, SparkInputTypes}; | ||
|
|
||
| /// Spark-compatible `timestamp_micros` expression. | ||
| /// | ||
| /// <https://spark.apache.org/docs/latest/api/sql/index.html#timestamp_micros> | ||
| #[user_doc( | ||
| doc_section(label = "Time and Date Functions"), | ||
| description = "Returns the timestamp that is `microseconds` microseconds after \ | ||
| 1970-01-01 00:00:00 UTC. Negative values precede the epoch. Unlike DataFusion's \ | ||
| `to_timestamp_micros`, only integer types whose full range fits in BIGINT are \ | ||
| accepted; floating-point, decimal and string inputs are rejected. The result has microsecond precision \ | ||
| and the session time zone (`datafusion.execution.time_zone`, default UTC). \ | ||
| The time zone affects display, not the stored microsecond count. Every BIGINT \ | ||
| value is valid, including its minimum and maximum. Returns NULL for NULL input.", | ||
| syntax_example = "timestamp_micros(microseconds)", | ||
| sql_example = r#"```sql | ||
| > SELECT timestamp_micros(0::BIGINT); | ||
| 1970-01-01T00:00:00Z | ||
| > SELECT timestamp_micros(-1::BIGINT); | ||
| 1969-12-31T23:59:59.999999Z | ||
| > SELECT timestamp_micros(NULL); | ||
| NULL | ||
| ```"#, | ||
| argument( | ||
| name = "microseconds", | ||
| description = "TINYINT, SMALLINT, INT or BIGINT count of microseconds since the UTC epoch. Arrow UInt8, UInt16 and UInt32 inputs are also accepted. UInt64 is rejected because its full range cannot fit in BIGINT. NULL returns NULL." | ||
| ) | ||
| )] | ||
| #[derive(Debug, PartialEq, Eq, Hash)] | ||
| pub struct SparkTimestampMicros { | ||
| input_types: SparkInputTypes, | ||
| timezone: Arc<str>, | ||
| } | ||
|
|
||
| impl Default for SparkTimestampMicros { | ||
| fn default() -> Self { | ||
| Self::new() | ||
| } | ||
| } | ||
|
|
||
| impl SparkTimestampMicros { | ||
| pub fn new() -> Self { | ||
| Self::new_with_config(&ConfigOptions::default()) | ||
| } | ||
|
|
||
| pub fn new_with_config(config: &ConfigOptions) -> Self { | ||
| Self { | ||
| input_types: SparkInputTypes::check_only( | ||
| vec![SparkInputType::Integral], | ||
| Volatility::Stable, | ||
| ), | ||
| timezone: config | ||
| .execution | ||
| .time_zone | ||
| .as_ref() | ||
| .map_or("UTC", |tz| tz.as_str()) | ||
| .into(), | ||
| } | ||
| } | ||
| } | ||
|
|
||
| impl ScalarUDFImpl for SparkTimestampMicros { | ||
| fn name(&self) -> &str { | ||
| "timestamp_micros" | ||
| } | ||
|
|
||
| fn signature(&self) -> &Signature { | ||
| self.input_types.signature() | ||
| } | ||
|
|
||
| fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> { | ||
| let types = self.input_types.coerce(self.name(), arg_types)?; | ||
| if types[0] == DataType::UInt64 { | ||
| return plan_err!( | ||
| "[DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] The first parameter of \ | ||
| `timestamp_micros` requires an integer type whose full range fits in \ | ||
| BIGINT, however it has the type \"UInt64\". SQLSTATE: 42K09" | ||
| ); | ||
| } | ||
| Ok(vec![DataType::Int64]) | ||
| } | ||
|
|
||
| fn return_type(&self, _arg_types: &[DataType]) -> Result<DataType> { | ||
| internal_err!("return_field_from_args should be used instead") | ||
| } | ||
|
|
||
| fn return_field_from_args(&self, args: ReturnFieldArgs) -> Result<FieldRef> { | ||
| let [input] = take_function_args(self.name(), args.arg_fields)?; | ||
| Ok(Arc::new(Field::new( | ||
| self.name(), | ||
| DataType::Timestamp(TimeUnit::Microsecond, Some(Arc::clone(&self.timezone))), | ||
| input.is_nullable(), | ||
| ))) | ||
| } | ||
|
|
||
| fn with_updated_config(&self, config: &ConfigOptions) -> Option<ScalarUDF> { | ||
| Some(ScalarUDF::from(Self::new_with_config(config))) | ||
| } | ||
|
|
||
| fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> { | ||
| let [microseconds] = take_function_args(self.name(), args.args)?; | ||
| // Both types store an i64 count, so this does not scale the value or | ||
| // convert it through a calendar representation (which has a smaller range). | ||
| microseconds.cast_to( | ||
| &DataType::Timestamp(TimeUnit::Microsecond, Some(Arc::clone(&self.timezone))), | ||
| None, | ||
| ) | ||
| } | ||
|
|
||
| fn documentation(&self) -> Option<&Documentation> { | ||
| self.doc() | ||
| } | ||
| } | ||
|
|
||
| #[cfg(test)] | ||
| mod tests { | ||
| use super::*; | ||
|
|
||
| #[test] | ||
| fn return_field_tracks_session_timezone_and_input_nullability() -> Result<()> { | ||
| for timezone in [None, Some("America/Los_Angeles")] { | ||
| let mut config = ConfigOptions::default(); | ||
| config.execution.time_zone = timezone.map(str::parse).transpose()?; | ||
| let function = SparkTimestampMicros::new() | ||
| .with_updated_config(&config) | ||
| .unwrap(); | ||
| for nullable in [false, true] { | ||
| let input = Arc::new(Field::new("micros", DataType::Int64, nullable)); | ||
| let field = function.inner().return_field_from_args(ReturnFieldArgs { | ||
| arg_fields: &[input], | ||
| scalar_arguments: &[None], | ||
| })?; | ||
| assert_eq!(field.is_nullable(), nullable); | ||
| assert_eq!( | ||
| field.data_type(), | ||
| &DataType::Timestamp( | ||
| TimeUnit::Microsecond, | ||
| Some(timezone.unwrap_or("UTC").into()), | ||
| ), | ||
| ); | ||
| } | ||
| } | ||
| Ok(()) | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree
SparkInputTypesisn't a fit as it stands, since it modelsImplicitCastInputTypesand would cast'1'.timestamp_millisandtimestamp_secondswill need the sameExpectsInputTypestreatment soon, though. What do you think about adding a check-only mode toimplicit_cast.rsso all three share it? That would also give this error the same Spark-shaped message the other functions use ("The first parameter oftimestamp_microsrequires the "INTEGRAL" type..."). If you'd rather keep this PR focused, a tracking issue linked here would work too.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Implemented in 1526fc0. Added
SparkInputTypes::check_onlyinimplicit_cast.rsand switchedtimestamp_microsto use it. It validates the declared input types without implicit casts, handles NULLs and dictionary layouts, and shares the Spark-style argument errors. The helper is available fortimestamp_millisandtimestamp_secondswhen those are implemented. Added coverage for validation in both ANSI modes and the error message. Workspace Clippy and all 516 SQL test runs pass.