Skip to content

Type numeric literals as Spark does in the sqllogictest files #97

Open

Description

@andygrove

With the Spark dialect, DataFusion types an unsuffixed decimal literal such as 3.14159 as DOUBLE, and an integer literal as BIGINT. Spark types them as DECIMAL, and as INT when the value fits. The SQL planner decides these types, and that is tracked upstream in apache/datafusion#27185. This issue tracks the work in this repository.

Impact

Many expected results in the .slt files were written with DataFusion's literal types, so they test a type Spark never produces for that query. The PySpark runner (#54) shows this:

Options

  1. Make the literal's type explicit in the tests, as 3.14159::DOUBLE, so that both engines test the same type. This is mechanical and can be done now.
  2. Enable datafusion.sql_parser.parse_float_as_decimal in the sqllogictest runner (crates/sqllogictest/tests/sqllogictests.rs), and possibly in SessionStateBuilderSpark::with_spark_features() for Spark-dialect users. A trial run made 11 of 245 files fail: try_sum, if, cast_float_to_timestamp, make_interval, stack, xxhash64, floor, hypot, negative, rint and format_string. Their expected results would need updating. It does not change integer literals.
  3. Use Spark's literal suffixes (3.14159D, 10Y, 1.5BD) once Spark, Databricks, Hive, Oracle: Parse numeric literal type suffixes datafusion-sqlparser-rs#2912 lands and DataFusion maps the suffixes to types. Spark reads these the same way.

Doing 1 now and 2 together with apache/datafusion#27185 means each expected result changes only once.

Activity

  1. andygrove commented on Nov 13, 2026

    @andygrove
    MemberAuthor

    The integer-literal half of this affects users as well as tests. date_add, date_sub and add_months only accept INT, SMALLINT and TINYINT, and space only accepts INT, as in Spark. An unsuffixed integer literal is BIGINT here, so the obvious call fails to plan:

    SELECT date_add(DATE '2024-01-01', 1);
    -- Error during planning: ... coercion from Date32, Int64 to the signature OneOf(Exact(Date32, Int8), Exact(Date32, Int16), Exact(Date32, Int32)) failed
    SELECT space(2);
    -- Error during planning: ... coercion from Int64 to the signature Uniform(1, [Int32, ...]) failed

    In Spark the literal is INT, so both work. Until literals are typed as Spark types them, the workaround is 1::INT. The docs in #105 and #106 say so. Widening the signatures to accept BIGINT would make these calls work, but Spark rejects a BIGINT column here, so it is better fixed in the literal typing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions