Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 76 additions & 0 deletions .github/workflows/cli.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership
# SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements

name: CLI

on:
push:
branches: [main]
pull_request:

permissions:
contents: read

jobs:
smoke-test:
name: datafusion-spark-cli smoke test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: rustup toolchain install stable --profile minimal --no-self-update && rustup default stable
- uses: Swatinem/rust-cache@v2
- run: cargo build -p datafusion-spark-cli
# A Spark function, and ANSI mode on by default and off with `--ansi false`.
- name: Run queries
run: |
set -euo pipefail
cli=target/debug/datafusion-spark-cli
$cli -q --format csv -c "SELECT sha2('Spark', 256) AS h" \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add a stdin REPL smoke test using CAST(1 AS LONG) and assert the result row? Current queries also pass with the generic dialect, so they do not verify Spark parsing.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in f0cf6de. CI pipes SELECT CAST(1 AS LONG) AS l; into the REPL and checks for the row 1.

| grep -x '529bc3b07127ecb7e53a4dcf1991d9152c24537d919178022b2c42657f79a26b'
if out=$($cli -q -c "SELECT abs(cast(-2147483648 AS int))" 2>&1); then
echo "expected an ARITHMETIC_OVERFLOW error with ANSI mode on" && exit 1
fi
grep -F '[ARITHMETIC_OVERFLOW]' <<< "$out"
$cli -q --ansi false --format csv -c "SELECT abs(cast(-2147483648 AS int)) AS a" \
| grep -x -- '-2147483648'
# SQL read from stdin by the REPL. `LONG` is a type in the Spark dialect only.
- name: Run the REPL from stdin
run: |
set -euo pipefail
echo "SELECT CAST(1 AS LONG) AS l;" \
| target/debug/datafusion-spark-cli -q --format csv | grep -x '1'
# `-f` and `-r` paths are relative to the working directory, not to `-p`, while
# paths in the SQL are relative to `-p`.
- name: Run files with a data path
run: |
set -euo pipefail
cli=$PWD/target/debug/datafusion-spark-cli
dir=$(mktemp -d) && cd "$dir" && mkdir data
echo "SELECT 'cwd' AS s;" > q.sql
echo "SELECT 'data' AS s;" > data/q.sql
$cli -q --format csv -p data -f q.sql | grep -x 'cwd'
echo "" | $cli -q --format csv -p data -r q.sql | grep -x 'cwd'
echo "SELECT 'stdin' AS s;" | $cli -q --format csv -p data -f /dev/stdin | grep -x 'stdin'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we also test that -p resolves the script from the launch directory while relative CSV paths resolve from the data directory? Use distinct CSV contents to verify both behaviors.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 224e924. The launch directory and data/ each have a t.csv with different contents, and data/ also has a decoy csv.sql. The script is run with -p data. CI checks that the script runs from the launch directory and reads data/t.csv.

printf 's\ncwd_csv\n' > t.csv
printf 's\ndata_csv\n' > data/t.csv
echo "SELECT s FROM 't.csv';" > csv.sql
echo "SELECT 'wrong script' AS s;" > data/csv.sql
out=$($cli -q --format csv -p data -f csv.sql)
grep -x 'data_csv' <<< "$out"
if grep -e 'cwd_csv' -e 'wrong script' <<< "$out"; then exit 1; fi
# Every file is opened before any runs, so a file that cannot be opened fails
# before an earlier file has written anything.
- name: Fail before running when a file cannot be opened
run: |
set -euo pipefail
cli=$PWD/target/debug/datafusion-spark-cli
dir=$(mktemp -d) && cd "$dir"
echo "COPY (SELECT 1 AS a) TO 'out.csv';" > first.sql
echo "SELECT 2 AS b;" > second.sql
chmod 000 second.sql
if out=$($cli -q -f first.sql second.sql 2>&1); then
echo "expected an error for the unreadable file" && exit 1
fi
grep -F "Cannot open file 'second.sql'" <<< "$out"
test ! -e out.csv
2 changes: 2 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ DataFusion expressions that emulate Apache Spark's behavior.

- `crates/spark` – the `datafusion-spark` crate
- `crates/sqllogictest` – `.slt` tests for the Spark functions
- `crates/cli` – `datafusion-spark-cli`, a SQL shell for trying the functions (not a default
workspace member, so build it with `-p datafusion-spark-cli`)

Before opening a PR, run `cargo fmt --all`, `cargo clippy --all-targets --all-features -- -D warnings`,
`cargo test -p datafusion-spark` and `cargo test -p datafusion-spark-sqllogictest`.
Expand Down
Loading