Skip to content

ci: run the PySpark runner on Spark 3.5 as well as 4.2 - #391

Open
andygrove wants to merge 5 commits into
apache:mainfrom
andygrove:pyspark-3.5
Open

andygrove wants to merge 5 commits into
apache:mainfrom
andygrove:pyspark-3.5

Conversation

@andygrove

@andygrove andygrove commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

  • None. This is a follow-up to the PySpark runner CI job.

Rationale for this change

CI checks the .slt files against Spark 4.2 only. Users also run Spark 3.5, and it behaves differently in places. This PR adds the latest 3.5 release (3.5.9) to CI, so a new difference from 3.5 shows up in the PR that adds it.

What changes are included in this PR?

  • CI: the spark job in pyspark-runner.yml is now a matrix over Spark 3.5 (requirements-spark-3.5.txt, pyspark==3.5.9) and 4.2 (requirements.txt), with fail-fast: false.
  • Required check: the PySpark runner jobs were not required, because the workflow's paths filters meant it never ran on most PRs, and a required check that never reports blocks the PR. The workflow now runs on every PR. A changes job (a git diff against the PR base, with no new actions) skips the unit-test and Spark jobs when nothing under crates/sqllogictest/scripts, crates/sqllogictest/test_files or the workflow changed. A PySpark runner job always runs and passes only if every job passed or was skipped. .asf.yaml now requires PySpark runner, which takes effect once this merges.
  • Runner support for Spark 3.5:
    • The runner accepts PySpark 3.5 and 4.2 (SUPPORTED_VERSIONS).
    • Spark 3.5 has no :: syntax, so on Spark 3 the runner rewrites expr::type to CAST(expr AS type). It follows Spark 4's precedence: :: binds to the literal, column, call, subscript or parenthesized expression before it, and a minus sign belongs to a number literal (-128::tinyint is CAST(-128 AS tinyint)). Without this rewrite, 1488 records failed to parse on 3.5. After it, 113 do, the same number as on 4.2.
    • PySpark 3.5 cannot read a schema that holds an interval (CANNOT_PARSE_DATATYPE). Those records are now skipped as a CalendarIntervalType value, as they already are on 4.2.
    • The runner works without pyspark.logger, which PySpark 3.5 does not have.
  • Version conditions on xfail markers: # xfail spark<4.0: <reason> applies only to the Spark versions that match. The operators are <, <=, >, >= and ==, compared on major.minor, and conditions separated by commas must all hold (spark>=4.0, <4.2). A record can have several markers, and the first that matches applies. Plain # xfail spark: still applies to every version. The runner stops with an error if it cannot read a condition.
  • Markers for the records where Spark 3.5 differs from Spark 4:
    • encode (7 records): Spark 3.5 accepts any Java charset (utf-32be, utf-32le) and replaces unmappable characters with ?. Spark 4 raises an error unless spark.sql.legacy.javaCharsets or spark.sql.legacy.codingErrorAction is set (checked on 4.2).
    • greatest (2 records): Spark 3.5 picks DECIMAL(38, 10) as the wider type of DECIMAL(38, 10) and DECIMAL(38, 0), where Spark 4 picks DECIMAL(38, 0).
    • to_json (1 record): Spark 3.5 drops the seconds of a local mean time offset (-07:52 instead of -07:52:58).
    • abs (2 records): Spark 3.5 agrees with DataFusion on large DECIMAL values, so these markers become spark>=4.0.

Are these changes tested?

  • New unit tests in test_pyspark_slt_runner.py cover the :: rewrite, version conditions, stacked markers, invalid conditions and markers for other versions. The tests pass on Python 3.9 and 3.12.
  • Full local runs, both exiting with status 0:
    • Spark 4.2.0: 3093 pass, 270 expected failures, 606 skipped, unchanged from main.
    • Spark 3.5.9: 2792 pass, 277 expected failures, 900 skipped. Most of the extra skips call functions that Spark 3.5 does not have, such as try_parse_url, make_time and is_valid_utf8.
  • cargo test -p datafusion-spark-sqllogictest passes. The .slt changes are comments only.

Are there any user-facing changes?

No. The changes are to CI and the test runner only. For contributors, PySpark runner becomes a required check.

The PySpark runner job now runs once for each supported Spark version,
3.5.9 and 4.2.0.

To run on Spark 3.5, the runner:
- accepts PySpark 3.5 and 4.2
- rewrites `expr::type` to `CAST(expr AS type)`, because Spark 3.5 has no
  `::` syntax
- skips interval results that PySpark 3.5 cannot read, as it already does
  on 4.2
- works without `pyspark.logger`, which PySpark 3.5 does not have

An `# xfail spark` comment can now take a version condition, such as
`# xfail spark<4.0: <reason>`, and a record can have several. The new
markers cover the records where Spark 3.5 differs from Spark 4: encode
charsets, greatest's wider DECIMAL type and to_json's offset seconds.
abs's DECIMAL markers now apply only to Spark 4.
A required check that never runs blocks a pull request, so the PySpark
runner workflow, which had `paths` filters, could not be required.

The workflow now runs on every pull request. A `changes` job skips the
unit-test and Spark jobs when nothing they test changed, and a
`PySpark runner` job always reports: it passes when those jobs pass or
are skipped. That job is now a required check in .asf.yaml.
- apply the :: rewrite in _run_record from the Spark version, in place
  of a double_colon flag passed through translate
- keep versions as (major, minor) tuples, with one parser
- match brackets with _matching_paren and a backward twin
- parse each file once
- diff the merge commit against its first parent, and fail the summary
  job with contains()

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove, this is a useful addition. Checking the .slt files against 3.5 in CI will catch version differences that matter to Comet. The CI structure with the changes job and the aggregate PySpark runner check looks right to me.


record = Record(path=path, line=i + 1, kind=words[0], xfail_reason=xfail_reason)
xfail_reason = None
record = Record(path=path, line=i + 1, kind=words[0], xfails=xfails)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Version conditions make it possible to write a marker that never applies on any Spark version CI runs. When the record passes, nothing reports it. For example, # xfail spark>=4.0, <4.2: x on a record that passes on both versions gives pass on 3.5.9 and on 4.2.0, not xpass. A marker shadowed by an earlier one that matches the same versions behaves the same way. That breaks the guarantee in run_record's docstring and the README that a stale comment gets reported.

Could we reject such markers at parse time? Then the error shows up before Spark starts, like a malformed condition does:

Suggested change
record = Record(path=path, line=i + 1, kind=words[0], xfails=xfails)
record = Record(path=path, line=i + 1, kind=words[0], xfails=xfails)
for x in xfails:
if all(record.xfail(v) is not x for v in SUPPORTED_VERSIONS):
raise SystemExit(
f"error: {_display_path(path)}:{i + 1}: `{x.text}` never applies, because no "
"supported Spark version matches it before an earlier marker does"
)

Every current .slt file parses with this check. test_xfail_spark_versions and the README example both use spark>=4.0, <4.2, which matches neither 3.5 nor 4.2, so they would need a different condition.

Comment on lines +871 to +872
"ALL", "AND", "ANY", "AS", "BETWEEN", "BY", "CASE", "DISTINCT", "ELSE",
"EXISTS", "FROM", "HAVING", "IN", "IS", "JOIN", "LIKE", "LIMIT", "NOT",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exists and any are Spark functions, but _operand_start treats them as keywords, so a :: after a call to either is rewritten wrong. SELECT exists(arr, x -> x > 1)::string FROM t becomes SELECT existsCAST((arr, x -> x > 1) AS string) FROM t, and any(a > 1)::string becomes anyCAST((a > 1) AS string). Spark 4.2 parses both originals as a cast of the call. Spark 3.5 raises PARSE_SYNTAX_ERROR on the rewritten SQL, and the runner turns a parse error into a skip. So such a record would silently go unchecked on 3.5. No current record hits this.

Neither word can come before a sign or a parenthesized expression that :: applies to, so they can leave the set:

Suggested change
"ALL", "AND", "ANY", "AS", "BETWEEN", "BY", "CASE", "DISTINCT", "ELSE",
"EXISTS", "FROM", "HAVING", "IN", "IS", "JOIN", "LIKE", "LIMIT", "NOT",
"ALL", "AND", "AS", "BETWEEN", "BY", "CASE", "DISTINCT", "ELSE",
"FROM", "HAVING", "IN", "IS", "JOIN", "LIKE", "LIMIT", "NOT",

Could you add both cases to test_double_colon_casts_for_spark_3?

Comment on lines +967 to +969
while k >= 2 and tokens[k - 1][1] == "." and tokens[k - 2][0] in ("word", "quoted"):
k -= 2
return k

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A field of a call or a subscript loses its base. SELECT struct(1 AS x).x::string becomes SELECT struct(1 AS x).CAST(x AS string), where Spark 4.2 casts struct(1 AS x).x. As with the keyword case, Spark 3.5 skips the record as a parse error. Could we continue into the operand before the .?

Suggested change
while k >= 2 and tokens[k - 1][1] == "." and tokens[k - 2][0] in ("word", "quoted"):
k -= 2
return k
while k >= 2 and tokens[k - 1][1] == "." and tokens[k - 2][0] in ("word", "quoted"):
k -= 2
if k >= 2 and tokens[k - 1][1] == "." and tokens[k - 2][1] in (")", "]"):
# A field of a call or subscript, such as `struct(1 AS x).x`
return _operand_start(tokens[: k - 1])
return k

With this, map(1, struct(2 AS y))[1].y::string also becomes CAST(map(1, struct(2 AS y))[1].y AS string).

Comment on lines +978 to +980
k += 1
j = _next_significant(tokens, k)
if j < len(tokens) and tokens[j][1] in ("(", "<"):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_type_end reads a < after any type name as the start of type parameters, so a comparison after a cast breaks. SELECT a::int < 5 AND b > 3 FROM t becomes SELECT CAST(a AS int < 5 AND b >) 3 FROM t, and SELECT a::int < 5 FROM t raises Skip("unbalanced parentheses"). Spark 4.2 parses the first as (CAST(a AS INT) < 5) AND (b > 3). On 3.5 either case ends as a skip. Comparisons after a cast are common in WHERE clauses, so I'd expect future tests to hit this. Only ARRAY, MAP, and STRUCT take <...>:

Suggested change
k += 1
j = _next_significant(tokens, k)
if j < len(tokens) and tokens[j][1] in ("(", "<"):
name = tokens[k][1].upper()
k += 1
j = _next_significant(tokens, k)
# After any other type, `<` is a comparison, as in `a::int < 5`.
if j < len(tokens) and (tokens[j][1] == "(" or (tokens[j][1] == "<" and name in ("ARRAY", "MAP", "STRUCT"))):

Could you add a::int < 5 and a::int<b::int to test_double_colon_casts_for_spark_3?

Decimal128(38, 0)

# The fractional digits that do not fit are rounded away
# xfail spark<4.0: intended: Spark 3.5 picks DECIMAL(38, 10) as the wider type and returns NULL for the value that does not fit, and datafusion-spark follows Spark 4

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reason doesn't describe what happens for this record. No value here fails to fit DECIMAL(38, 10), and Spark 3.5.9 returns 1.5000000000 where the file expects 2. The reason on line 102 is right for its record: the DECIMAL(38, 0) value becomes NULL, and greatest returns 1.0000000000.

Suggested change
# xfail spark<4.0: intended: Spark 3.5 picks DECIMAL(38, 10) as the wider type and returns NULL for the value that does not fit, and datafusion-spark follows Spark 4
# xfail spark<4.0: intended: Spark 3.5 picks DECIMAL(38, 10) as the wider type and keeps the fraction, and datafusion-spark follows Spark 4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants