You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Type numeric literals as Spark does in the sqllogictest files #97
With the Spark dialect, DataFusion types an unsuffixed decimal literal such as 3.14159 as DOUBLE, and an integer literal as BIGINT. Spark types them as DECIMAL, and as INT when the value fits. The SQL planner decides these types, and that is tracked upstream in apache/datafusion#27185. This issue tracks the work in this repository.
Impact
Many expected results in the .slt files were written with DataFusion's literal types, so they test a type Spark never produces for that query. The PySpark runner (#54) shows this:
xxhash64(1) hashes a BIGINT in DataFusion and an INT in Spark.
Options
Make the literal's type explicit in the tests, as 3.14159::DOUBLE, so that both engines test the same type. This is mechanical and can be done now.
Enable datafusion.sql_parser.parse_float_as_decimal in the sqllogictest runner (crates/sqllogictest/tests/sqllogictests.rs), and possibly in SessionStateBuilderSpark::with_spark_features() for Spark-dialect users. A trial run made 11 of 245 files fail: try_sum, if, cast_float_to_timestamp, make_interval, stack, xxhash64, floor, hypot, negative, rint and format_string. Their expected results would need updating. It does not change integer literals.
The integer-literal half of this affects users as well as tests. date_add, date_sub and add_months only accept INT, SMALLINT and TINYINT, and space only accepts INT, as in Spark. An unsuffixed integer literal is BIGINT here, so the obvious call fails to plan:
SELECT date_add(DATE'2024-01-01', 1);
-- Error during planning: ... coercion from Date32, Int64 to the signature OneOf(Exact(Date32, Int8), Exact(Date32, Int16), Exact(Date32, Int32)) failedSELECT space(2);
-- Error during planning: ... coercion from Int64 to the signature Uniform(1, [Int32, ...]) failed
In Spark the literal is INT, so both work. Until literals are typed as Spark types them, the workaround is 1::INT. The docs in #105 and #106 say so. Widening the signatures to accept BIGINT would make these calls work, but Spark rejects a BIGINT column here, so it is better fixed in the literal typing.
Activity
andygrove commentedon Nov 13, 2026
The integer-literal half of this affects users as well as tests.
date_add,date_subandadd_monthsonly accept INT, SMALLINT and TINYINT, andspaceonly accepts INT, as in Spark. An unsuffixed integer literal is BIGINT here, so the obvious call fails to plan:In Spark the literal is INT, so both work. Until literals are typed as Spark types them, the workaround is
1::INT. The docs in #105 and #106 say so. Widening the signatures to accept BIGINT would make these calls work, but Spark rejects a BIGINT column here, so it is better fixed in the literal typing.docs(spark): link known date function gaps to their issues
docs(spark): link space integer literal typing to apache#97
docs(spark): document date arithmetic and field functions with #[user…
docs(spark): document string functions with #[user_doc] (#105)
test: treat a different sign on zero as a real difference in the PySp…
feat(spark): add Spark-compatible array_max and array_min (#331)
test: mark slt records that differ from Spark with an xfail comment (#…