Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .asf.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,8 @@ github:
main:
required_pull_request_reviews:
required_approving_review_count: 1
# Job names from .github/workflows. The PySpark runner jobs are left out
# because they only run when their paths change, and a required check
# that does not run blocks the pull request.
# Job names from .github/workflows. Each of these jobs runs on every
# pull request, because a required check that does not run blocks it.
required_status_checks:
contexts:
- "cargo fmt"
Expand All @@ -47,6 +46,7 @@ github:
- "Run sqllogictests"
- "Apache RAT"
- "ASF allowlist check"
- "PySpark runner"
pull_requests:
# enable updating head branches of pull requests
allow_update_branch: true
Expand Down
66 changes: 54 additions & 12 deletions .github/workflows/pyspark-runner.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,25 +4,47 @@

name: PySpark runner

# The workflow runs on every pull request, so that its `PySpark runner` job
# always reports and can be a required check. A `paths` filter would leave the
# check pending, and block the pull request, when the paths do not change.
# Instead the `changes` job skips the Spark jobs when nothing they test changed.
on:
push:
branches: [main]
paths:
- "crates/sqllogictest/scripts/**"
- "crates/sqllogictest/test_files/**"
- ".github/workflows/pyspark-runner.yml"
pull_request:
paths:
- "crates/sqllogictest/scripts/**"
- "crates/sqllogictest/test_files/**"
- ".github/workflows/pyspark-runner.yml"

permissions:
contents: read

jobs:
changes:
name: Check for PySpark runner changes
runs-on: ubuntu-latest
outputs:
runner: ${{ steps.diff.outputs.runner }}
steps:
# A pull request checks out a merge commit, whose first parent is main.
- uses: actions/checkout@v4
with:
fetch-depth: 2
- id: diff
env:
EVENT: ${{ github.event_name }}
run: |
if [ "$EVENT" != pull_request ] ||
git diff --name-only HEAD^1 HEAD -- \
crates/sqllogictest/scripts \
crates/sqllogictest/test_files \
.github/workflows/pyspark-runner.yml | grep -q .; then
echo runner=true >> "$GITHUB_OUTPUT"
else
echo runner=false >> "$GITHUB_OUTPUT"
fi

unit-tests:
name: PySpark runner unit tests
needs: changes
if: needs.changes.outputs.runner == 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
Expand All @@ -32,8 +54,18 @@ jobs:
- run: python3 -m unittest discover -s crates/sqllogictest/scripts

spark:
name: Run the sqllogictests on Spark
name: Run the sqllogictests on Spark ${{ matrix.spark }}
needs: changes
if: needs.changes.outputs.runner == 'true'
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
include:
- spark: "3.5"
requirements: requirements-spark-3.5.txt
- spark: "4.2"
requirements: requirements.txt
steps:
- uses: actions/checkout@v4
- uses: actions/setup-java@v4
Expand All @@ -44,8 +76,18 @@ jobs:
with:
python-version: "3.12"
cache: pip
cache-dependency-path: crates/sqllogictest/scripts/requirements.txt
- run: pip install -r crates/sqllogictest/scripts/requirements.txt
cache-dependency-path: crates/sqllogictest/scripts/${{ matrix.requirements }}
- run: pip install -r crates/sqllogictest/scripts/${{ matrix.requirements }}
# Exits with status 1 if any record differs from Spark without an
# `# xfail spark:` comment, or has one but now passes.
# `# xfail spark:` comment for this Spark version, or has one but now passes.
- run: python crates/sqllogictest/scripts/pyspark_slt_runner.py

# The required check. It passes when the jobs above pass or are skipped.
result:
name: PySpark runner
needs: [changes, unit-tests, spark]
if: always()
runs-on: ubuntu-latest
steps:
- if: contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled')
run: exit 1
41 changes: 35 additions & 6 deletions crates/sqllogictest/scripts/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,15 @@ SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one

`pyspark_slt_runner.py` runs the `.slt` files in
[`crates/sqllogictest/test_files`](../test_files)
on Apache Spark 4.2 and reports every query where Spark's result differs from
the expected result in the file.
on Apache Spark and reports every query where Spark's result differs from the
expected result in the file. It supports Spark 3.5 and 4.2.

The expected results in those files come from DataFusion, and nothing else
checks them against Spark. A difference means that DataFusion and Spark
disagree, or that the test expects the wrong value. CI runs the runner on
every pull request that changes an `.slt` file or the runner, and fails if it
reports a difference that has no `# xfail spark:` comment (see
every pull request that changes an `.slt` file or the runner, once for each
supported Spark version, and fails if it reports a difference that has no
`# xfail spark:` comment for that version (see
[Known differences](#known-differences)) or a comment on a record that now
passes.

Expand All @@ -29,8 +30,16 @@ python3 -m venv /tmp/pyspark-4.2
/tmp/pyspark-4.2/bin/pip install -r crates/sqllogictest/scripts/requirements.txt
```

The runner ignores `SPARK_HOME`, so it always uses the Spark bundled with this
PySpark.
For Spark 3.5, install `requirements-spark-3.5.txt` into a second virtual
environment, and run the runner with that environment's Python:

```shell
python3 -m venv /tmp/pyspark-3.5
/tmp/pyspark-3.5/bin/pip install -r crates/sqllogictest/scripts/requirements-spark-3.5.txt
```

The runner ignores `SPARK_HOME`, so it always uses the Spark bundled with the
installed PySpark, and it checks that this is a supported version.

## Usage

Expand Down Expand Up @@ -78,6 +87,22 @@ SELECT round(arrow_cast(2.5, 'Decimal32(9, 1)'), 0::int);
3.0
```

A comparison after `spark` limits the comment to some Spark versions, which it
compares by major and minor version. Several comparisons separated by commas
must all hold:

```text
# xfail spark<4.0: intended: Spark 3.5 accepts any Java charset name, which Spark 4 allows only with spark.sql.legacy.javaCharsets
statement error INVALID_PARAMETER_VALUE\.CHARSET
SELECT encode('A'::string, 'utf-32be'::string);
```

The operators are `<`, `<=`, `>`, `>=` and `==`, as in `spark>=4.0, <=4.2`. A
record can have several comments, one line each, and the first that matches
the Spark version applies. The runner stops with an error if it cannot read a
version condition, or if a comment applies to none of the Spark versions it
supports because no version matches it before an earlier comment does.

The reason links an issue, or says what differs and why: a bug in
datafusion-spark, a difference that comes from how the test is written, or a
limitation of this runner. A marked record that differs from Spark in any way
Expand Down Expand Up @@ -116,6 +141,10 @@ are DataFusion's defaults. These settings in the files are passed on to Spark:
Spark 4.2 parses most of the SQL as written, including `::` casts. The runner
makes only these changes:

- For Spark 3.5, which has no `::` syntax, `expr::type` becomes
`CAST(expr AS type)`. As in Spark 4, `::` applies to the literal, column,
call, subscript or parenthesized expression before it, and a minus sign
belongs to a number, so `-1::int` becomes `CAST(-1 AS int)`.
- `CREATE TABLE` becomes `CREATE TEMPORARY VIEW`, and `DROP TABLE` becomes
`DROP VIEW`
- PostgreSQL type names that Spark does not accept are renamed, for example
Expand Down
Loading
Loading