Repository navigation
Conversation
The PySpark runner job now runs once for each supported Spark version, 3.5.9 and 4.2.0. To run on Spark 3.5, the runner: - accepts PySpark 3.5 and 4.2 - rewrites `expr::type` to `CAST(expr AS type)`, because Spark 3.5 has no `::` syntax - skips interval results that PySpark 3.5 cannot read, as it already does on 4.2 - works without `pyspark.logger`, which PySpark 3.5 does not have An `# xfail spark` comment can now take a version condition, such as `# xfail spark<4.0: <reason>`, and a record can have several. The new markers cover the records where Spark 3.5 differs from Spark 4: encode charsets, greatest's wider DECIMAL type and to_json's offset seconds. abs's DECIMAL markers now apply only to Spark 4.
A required check that never runs blocks a pull request, so the PySpark runner workflow, which had `paths` filters, could not be required. The workflow now runs on every pull request. A `changes` job skips the unit-test and Spark jobs when nothing they test changed, and a `PySpark runner` job always reports: it passes when those jobs pass or are skipped. That job is now a required check in .asf.yaml.
- apply the :: rewrite in _run_record from the Spark version, in place of a double_colon flag passed through translate - keep versions as (major, minor) tuples, with one parser - match brackets with _matching_paren and a backward twin - parse each file once - diff the merge commit against its first parent, and fail the summary job with contains()
mbutrovich
left a comment
There was a problem hiding this comment.
Thanks @andygrove, this is a useful addition. Checking the .slt files against 3.5 in CI will catch version differences that matter to Comet. The CI structure with the changes job and the aggregate PySpark runner check looks right to me.
|
|
||
| record = Record(path=path, line=i + 1, kind=words[0], xfail_reason=xfail_reason) | ||
| xfail_reason = None | ||
| record = Record(path=path, line=i + 1, kind=words[0], xfails=xfails) |
There was a problem hiding this comment.
Version conditions make it possible to write a marker that never applies on any Spark version CI runs. When the record passes, nothing reports it. For example, # xfail spark>=4.0, <4.2: x on a record that passes on both versions gives pass on 3.5.9 and on 4.2.0, not xpass. A marker shadowed by an earlier one that matches the same versions behaves the same way. That breaks the guarantee in run_record's docstring and the README that a stale comment gets reported.
Could we reject such markers at parse time? Then the error shows up before Spark starts, like a malformed condition does:
| record = Record(path=path, line=i + 1, kind=words[0], xfails=xfails) | |
| record = Record(path=path, line=i + 1, kind=words[0], xfails=xfails) | |
| for x in xfails: | |
| if all(record.xfail(v) is not x for v in SUPPORTED_VERSIONS): | |
| raise SystemExit( | |
| f"error: {_display_path(path)}:{i + 1}: `{x.text}` never applies, because no " | |
| "supported Spark version matches it before an earlier marker does" | |
| ) |
Every current .slt file parses with this check. test_xfail_spark_versions and the README example both use spark>=4.0, <4.2, which matches neither 3.5 nor 4.2, so they would need a different condition.
| "ALL", "AND", "ANY", "AS", "BETWEEN", "BY", "CASE", "DISTINCT", "ELSE", | ||
| "EXISTS", "FROM", "HAVING", "IN", "IS", "JOIN", "LIKE", "LIMIT", "NOT", |
There was a problem hiding this comment.
exists and any are Spark functions, but _operand_start treats them as keywords, so a :: after a call to either is rewritten wrong. SELECT exists(arr, x -> x > 1)::string FROM t becomes SELECT existsCAST((arr, x -> x > 1) AS string) FROM t, and any(a > 1)::string becomes anyCAST((a > 1) AS string). Spark 4.2 parses both originals as a cast of the call. Spark 3.5 raises PARSE_SYNTAX_ERROR on the rewritten SQL, and the runner turns a parse error into a skip. So such a record would silently go unchecked on 3.5. No current record hits this.
Neither word can come before a sign or a parenthesized expression that :: applies to, so they can leave the set:
| "ALL", "AND", "ANY", "AS", "BETWEEN", "BY", "CASE", "DISTINCT", "ELSE", | |
| "EXISTS", "FROM", "HAVING", "IN", "IS", "JOIN", "LIKE", "LIMIT", "NOT", | |
| "ALL", "AND", "AS", "BETWEEN", "BY", "CASE", "DISTINCT", "ELSE", | |
| "FROM", "HAVING", "IN", "IS", "JOIN", "LIKE", "LIMIT", "NOT", |
Could you add both cases to test_double_colon_casts_for_spark_3?
| while k >= 2 and tokens[k - 1][1] == "." and tokens[k - 2][0] in ("word", "quoted"): | ||
| k -= 2 | ||
| return k |
There was a problem hiding this comment.
A field of a call or a subscript loses its base. SELECT struct(1 AS x).x::string becomes SELECT struct(1 AS x).CAST(x AS string), where Spark 4.2 casts struct(1 AS x).x. As with the keyword case, Spark 3.5 skips the record as a parse error. Could we continue into the operand before the .?
| while k >= 2 and tokens[k - 1][1] == "." and tokens[k - 2][0] in ("word", "quoted"): | |
| k -= 2 | |
| return k | |
| while k >= 2 and tokens[k - 1][1] == "." and tokens[k - 2][0] in ("word", "quoted"): | |
| k -= 2 | |
| if k >= 2 and tokens[k - 1][1] == "." and tokens[k - 2][1] in (")", "]"): | |
| # A field of a call or subscript, such as `struct(1 AS x).x` | |
| return _operand_start(tokens[: k - 1]) | |
| return k |
With this, map(1, struct(2 AS y))[1].y::string also becomes CAST(map(1, struct(2 AS y))[1].y AS string).
| k += 1 | ||
| j = _next_significant(tokens, k) | ||
| if j < len(tokens) and tokens[j][1] in ("(", "<"): |
There was a problem hiding this comment.
_type_end reads a < after any type name as the start of type parameters, so a comparison after a cast breaks. SELECT a::int < 5 AND b > 3 FROM t becomes SELECT CAST(a AS int < 5 AND b >) 3 FROM t, and SELECT a::int < 5 FROM t raises Skip("unbalanced parentheses"). Spark 4.2 parses the first as (CAST(a AS INT) < 5) AND (b > 3). On 3.5 either case ends as a skip. Comparisons after a cast are common in WHERE clauses, so I'd expect future tests to hit this. Only ARRAY, MAP, and STRUCT take <...>:
| k += 1 | |
| j = _next_significant(tokens, k) | |
| if j < len(tokens) and tokens[j][1] in ("(", "<"): | |
| name = tokens[k][1].upper() | |
| k += 1 | |
| j = _next_significant(tokens, k) | |
| # After any other type, `<` is a comparison, as in `a::int < 5`. | |
| if j < len(tokens) and (tokens[j][1] == "(" or (tokens[j][1] == "<" and name in ("ARRAY", "MAP", "STRUCT"))): |
Could you add a::int < 5 and a::int<b::int to test_double_colon_casts_for_spark_3?
| Decimal128(38, 0) | ||
|
|
||
| # The fractional digits that do not fit are rounded away | ||
| # xfail spark<4.0: intended: Spark 3.5 picks DECIMAL(38, 10) as the wider type and returns NULL for the value that does not fit, and datafusion-spark follows Spark 4 |
There was a problem hiding this comment.
This reason doesn't describe what happens for this record. No value here fails to fit DECIMAL(38, 10), and Spark 3.5.9 returns 1.5000000000 where the file expects 2. The reason on line 102 is right for its record: the DECIMAL(38, 0) value becomes NULL, and greatest returns 1.0000000000.
| # xfail spark<4.0: intended: Spark 3.5 picks DECIMAL(38, 10) as the wider type and returns NULL for the value that does not fit, and datafusion-spark follows Spark 4 | |
| # xfail spark<4.0: intended: Spark 3.5 picks DECIMAL(38, 10) as the wider type and keeps the fraction, and datafusion-spark follows Spark 4 |
Which issue does this PR close?
Rationale for this change
CI checks the
.sltfiles against Spark 4.2 only. Users also run Spark 3.5, and it behaves differently in places. This PR adds the latest 3.5 release (3.5.9) to CI, so a new difference from 3.5 shows up in the PR that adds it.What changes are included in this PR?
sparkjob inpyspark-runner.ymlis now a matrix over Spark 3.5 (requirements-spark-3.5.txt,pyspark==3.5.9) and 4.2 (requirements.txt), withfail-fast: false.pathsfilters meant it never ran on most PRs, and a required check that never reports blocks the PR. The workflow now runs on every PR. Achangesjob (agit diffagainst the PR base, with no new actions) skips the unit-test and Spark jobs when nothing undercrates/sqllogictest/scripts,crates/sqllogictest/test_filesor the workflow changed. APySpark runnerjob always runs and passes only if every job passed or was skipped..asf.yamlnow requiresPySpark runner, which takes effect once this merges.SUPPORTED_VERSIONS).::syntax, so on Spark 3 the runner rewritesexpr::typetoCAST(expr AS type). It follows Spark 4's precedence:::binds to the literal, column, call, subscript or parenthesized expression before it, and a minus sign belongs to a number literal (-128::tinyintisCAST(-128 AS tinyint)). Without this rewrite, 1488 records failed to parse on 3.5. After it, 113 do, the same number as on 4.2.CANNOT_PARSE_DATATYPE). Those records are now skipped as aCalendarIntervalTypevalue, as they already are on 4.2.pyspark.logger, which PySpark 3.5 does not have.# xfail spark<4.0: <reason>applies only to the Spark versions that match. The operators are<,<=,>,>=and==, compared on major.minor, and conditions separated by commas must all hold (spark>=4.0, <4.2). A record can have several markers, and the first that matches applies. Plain# xfail spark:still applies to every version. The runner stops with an error if it cannot read a condition.encode(7 records): Spark 3.5 accepts any Java charset (utf-32be,utf-32le) and replaces unmappable characters with?. Spark 4 raises an error unlessspark.sql.legacy.javaCharsetsorspark.sql.legacy.codingErrorActionis set (checked on 4.2).greatest(2 records): Spark 3.5 picksDECIMAL(38, 10)as the wider type ofDECIMAL(38, 10)andDECIMAL(38, 0), where Spark 4 picksDECIMAL(38, 0).to_json(1 record): Spark 3.5 drops the seconds of a local mean time offset (-07:52instead of-07:52:58).abs(2 records): Spark 3.5 agrees with DataFusion on largeDECIMALvalues, so these markers becomespark>=4.0.Are these changes tested?
test_pyspark_slt_runner.pycover the::rewrite, version conditions, stacked markers, invalid conditions and markers for other versions. The tests pass on Python 3.9 and 3.12.main.try_parse_url,make_timeandis_valid_utf8.cargo test -p datafusion-spark-sqllogictestpasses. The.sltchanges are comments only.Are there any user-facing changes?
No. The changes are to CI and the test runner only. For contributors,
PySpark runnerbecomes a required check.