Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
57cc4cb
spark: establish Joom Spark 4.2.1 and Delta baseline
kotlovs Nov 17, 2026
5a9cbf8
native: vendor pristine datafusion-physical-plan 55.2.0
kotlovs Nov 17, 2026
1322919
native: test vendored DataFusion in the Comet workspace
kotlovs Nov 17, 2026
d354026
ci: exclude generated Spark trees from suite inventory
kotlovs Nov 17, 2026
0710508
release: build and validate Joom Comet for Spark 4.2.1
kotlovs Nov 17, 2026
92e97e4
test: import shuffle exchange in Spark 4.2.1 fixture
kotlovs Nov 17, 2026
ed5d78e
release: handle optional Maven flags in macOS Bash
kotlovs Nov 17, 2026
87f58a8
test: run legacy ANSI overflow cases on Spark 4.2
kotlovs Nov 17, 2026
a41d3e9
ci: use standard Ubuntu runners for Joom PR gates
kotlovs Nov 17, 2026
f8223f0
cast: match Spark tiny-decimal rescaling fixes
kotlovs Nov 17, 2026
3c1e8b7
parquet: reject duplicate projected field selection
kotlovs Nov 17, 2026
99ed2f6
union: enforce known partition layouts during execution
kotlovs Nov 17, 2026
b642976
test: preserve Spark SQL contracts with Comet plans
kotlovs Nov 17, 2026
72758d3
test(delta): give deletion-vector fixtures an explicit offset
kotlovs Nov 17, 2026
d1c8534
ci: retry transient parent POM resolution before builds
kotlovs Nov 17, 2026
43af9be
union: preserve partitioning policy and bindings through AQE copies
kotlovs Nov 17, 2026
6277197
parquet: assert typed read errors from duplicate-name prevalidation
kotlovs Nov 17, 2026
cced071
release: build and smoke-test RC artifacts on native Linux runners
kotlovs Nov 18, 2026
c04d2d0
docs: finalize G01 migration decisions and English handoff
kotlovs Nov 18, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/actions/java-test/action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,10 @@ runs:
# cache) by setup-builder / setup-macos-builder via maven-bootstrap, which
# every caller of this action runs first.

- name: Resolve Maven model before tests
shell: bash
run: dev/ci/prepare-maven-model.sh ${{ inputs.maven_opts }}

- name: Run all tests
shell: bash
if: ${{ inputs.suites == '' }}
Expand Down
22 changes: 17 additions & 5 deletions .github/actions/rust-test/action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -37,13 +37,16 @@ runs:
shell: bash
run: |
cd native
cargo clippy --color=never --all-targets --workspace -- -D warnings
# Published DataFusion 55.2 uses fetch_update, deprecated by Rust 1.100.
# Preserve its sources and lint every Comet member strictly; vendor unit
# tests below remain explicit and mandatory.
cargo clippy --locked --color=never --all-targets --workspace --exclude datafusion-physical-plan --no-deps -- -D warnings

- name: Check native Python UDF feature
shell: bash
run: |
cd native
cargo clippy --color=never -p datafusion-comet --all-targets --features python-udf -- -D warnings
cargo clippy --locked --color=never -p datafusion-comet --all-targets --features python-udf --no-deps -- -D warnings

- name: Check compilation
shell: bash
Expand Down Expand Up @@ -74,8 +77,7 @@ runs:
- name: Build common module (pre-requisite for Rust tests)
shell: bash
run: |
cd common
../mvnw -B clean compile -DskipTests
./mvnw -B compile -DskipTests

- name: Install nextest
shell: bash
Expand Down Expand Up @@ -106,6 +108,16 @@ runs:
export LD_LIBRARY_PATH=${JAVA_HOME}/lib/server:${LD_LIBRARY_PATH}
RUST_BACKTRACE=1 cargo nextest run -p datafusion-comet --lib --features python-udf python_udf

- name: Test vendored DataFusion with the Comet workspace graph
shell: bash
run: |
cd native
export LD_LIBRARY_PATH=${JAVA_HOME}/lib/server:${LD_LIBRARY_PATH}
# Building both packages preserves Comet's feature unification; the filter
# runs the vendor tests explicitly instead of relying on default-members.
cargo nextest run --locked -p datafusion-comet -p datafusion-physical-plan --lib -E 'package(datafusion-physical-plan)'
cargo nextest run --locked -p datafusion-comet -p datafusion-physical-plan --lib --features datafusion-physical-plan/test_utils -E 'package(datafusion-physical-plan) & test(statistics::)'

# The steps above lint and test the accounting allocator over the system allocator. Lint and
# test it over jemalloc too, since that is a different allocator backend and nothing else in CI
# builds it.
Expand All @@ -114,5 +126,5 @@ runs:
run: |
cd native
export LD_LIBRARY_PATH=${JAVA_HOME}/lib/server:${LD_LIBRARY_PATH}
cargo clippy --color=never -p datafusion-comet --all-targets --features jemalloc -- -D warnings
cargo clippy --locked --color=never -p datafusion-comet --all-targets --features jemalloc --no-deps -- -D warnings
RUST_BACKTRACE=1 cargo nextest run -p datafusion-comet --lib --features jemalloc alloc_accounting
8 changes: 4 additions & 4 deletions .github/workflows/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,15 +91,15 @@ tiers partition the list and that the `pr` tier is exactly the default profile.
|
v
+-----------------------+
| preflight | ubuntu-slim
| preflight | ubuntu-24.04
| (RAT, prettier, |
| missing-suites, |
| actionlint) |
+-----------+-----------+
| on success
v
+-----------------------+
| changes | ubuntu-slim
| changes | ubuntu-24.04
| (compute-changes.py: |
| one boolean per |
| heavy job) |
Expand Down Expand Up @@ -137,14 +137,14 @@ tiers partition the list and that the `pr` tier is exactly the default profile.
+-----------------------------------+-----------------------------------+
v
+-----------------------+
| required_checks | ubuntu-slim
| required_checks | ubuntu-24.04
| one flat name that |
| is safe to require |
+-----------+-----------+
| schedule only, when red
v
+-----------------------+
| nightly_report | ubuntu-slim
| nightly_report | ubuntu-24.04
| opens or comments on |
| a ci-nightly-failure |
| issue |
Expand Down
39 changes: 27 additions & 12 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -124,14 +124,14 @@ jobs:
# required-status-check evaluation read. Guarding this job on the label name
# therefore let any non-gating label overwrite the commit run's real
# `Preflight` verdict with `skipped` (issue #5007). Running it unconditionally
# costs about a minute of ubuntu-slim time per label event and keeps the
# costs about a minute of runner time per label event and keeps the
# reported verdict truthful. Skipping the redundant work is the heavy jobs'
# job; POLICY in dev/ci/compute-changes.py drops every job a `labeled` event
# does not gate.
# ---------------------------------------------------------------------------
preflight:
name: Preflight
runs-on: ubuntu-slim
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v7

Expand Down Expand Up @@ -164,10 +164,9 @@ jobs:
- name: Install mermaid-cli
run: |
set -eux
# ubuntu-slim carries none of Chrome's shared libraries, so the browser downloads
# fine and then dies with `libatk-1.0.so.0: cannot open shared object file`. These
# are what chrome-headless-shell actually links against; libgtk-3 pulls in most of
# the rest (atk, cairo, pango, gdk-pixbuf) as dependencies.
# Install Chrome's runtime libraries explicitly so the browser setup does not
# depend on which packages happen to be preinstalled on the runner.
# libgtk-3 pulls in atk, cairo, pango and gdk-pixbuf as dependencies.
sudo apt-get update -qq
sudo apt-get install -y -qq --no-install-recommends \
libgtk-3-0t64 libnss3 libasound2t64 libgbm1 libatk-bridge2.0-0t64 \
Expand Down Expand Up @@ -200,6 +199,9 @@ jobs:
- name: Check Iceberg write report summary
run: python3 dev/ci/test-summarize-iceberg-writes.py

- name: Check model-only Maven retry
run: python3 dev/ci/test-maven-model.py

- name: Check CI config invariants
run: python3 dev/ci/check-ci-config.py

Expand Down Expand Up @@ -237,7 +239,7 @@ jobs:
changes:
name: Detect changes
needs: preflight
runs-on: ubuntu-slim
runs-on: ubuntu-24.04
# `actions: read` is for dev/ci/nightly-base.py, which lists previous
# scheduled runs to pick the nightly's diff base.
permissions:
Expand Down Expand Up @@ -266,6 +268,7 @@ jobs:
delta_3_5: ${{ steps.compute.outputs.delta_3_5 }}
delta_4_0: ${{ steps.compute.outputs.delta_4_0 }}
delta_4_1: ${{ steps.compute.outputs.delta_4_1 }}
delta_4_2: ${{ steps.compute.outputs.delta_4_2 }}
steps:
- uses: actions/checkout@v7
with:
Expand Down Expand Up @@ -501,14 +504,25 @@ jobs:
needs: changes
permissions:
contents: read
# Nightly by default; PRs need the `run-delta-tests` label. No Spark 4.2
# caller: the contrib profile is dormant until a Delta release supports it.
# Nightly by default; PRs need the `run-delta-tests` label.
if: needs.changes.outputs.delta_4_1 == 'true'
uses: ./.github/workflows/delta_contrib_test.yml
with:
spark: '4.1'
java: '17'

delta_4_2:
name: Delta Contrib Tests (Spark 4.2)
needs: changes
permissions:
contents: read
# Delta 4.4.1 publishes a dedicated Spark 4.2 artifact.
if: needs.changes.outputs.delta_4_2 == 'true'
uses: ./.github/workflows/delta_contrib_test.yml
with:
spark: '4.2'
java: '17'

spark_4_0:
name: Spark SQL Tests (Spark 4.0)
needs: changes
Expand Down Expand Up @@ -552,7 +566,7 @@ jobs:
uses: ./.github/workflows/spark_sql_test_reusable.yml
with:
spark-short: '4.2'
spark-full: '4.2.0'
spark-full: '4.2.1'
java: 17

iceberg_1_8:
Expand Down Expand Up @@ -659,14 +673,15 @@ jobs:
- delta_3_5
- delta_4_0
- delta_4_1
- delta_4_2
- spark_4_0
- spark_4_1
- spark_4_2
- iceberg_1_8
- iceberg_1_9
- iceberg_1_10
- iceberg_1_11
runs-on: ubuntu-slim
runs-on: ubuntu-24.04
steps:
- name: Summarize upstream results
env:
Expand Down Expand Up @@ -703,7 +718,7 @@ jobs:
permissions:
actions: read
issues: write
runs-on: ubuntu-slim
runs-on: ubuntu-24.04
steps:
- uses: actions/github-script@v9
with:
Expand Down
13 changes: 13 additions & 0 deletions .github/workflows/delta_build_gate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,19 @@ jobs:
rust-version: ${{ env.RUST_VERSION }}
jdk-version: 17

- name: Restore Maven dependencies
uses: actions/cache/restore@v6
with:
path: |
~/.m2/repository
/root/.m2/repository
key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-delta-build-gate
restore-keys: |
${{ runner.os }}-java-maven-

- name: Resolve Maven model before build-gate checks
run: dev/ci/prepare-maven-model.sh

- name: Run dev/verify-contrib-delta-gate.sh
run: |
dev/verify-contrib-delta-gate.sh
4 changes: 2 additions & 2 deletions .github/workflows/delta_contrib_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -113,11 +113,11 @@ jobs:
- name: Install Comet core jars
run: |
# The delta profile is what produces the spark test-jar the contrib suites depend on.
./mvnw -B -q -Prelease -Pspark-${{ inputs.spark }},delta install -pl common,spark -DskipTests -Dspotless.check.skip=true
./mvnw -B -q -Prelease -Pspark-${{ inputs.spark }},delta install -DskipTests -Dspotless.check.skip=true

- name: Run Delta contrib test suites
run: |
SPARK_HOME=$(pwd) COMET_CONF_DIR=$(pwd)/conf ./mvnw -B -Prelease -Pspark-${{ inputs.spark }},delta test -pl contrib/delta-spark
SPARK_HOME=$(pwd) COMET_CONF_DIR=$(pwd)/conf ./mvnw -B -Prelease -Pspark-${{ inputs.spark }},delta verify -Dtest=none -DwildcardSuites=org.apache.comet.contrib.delta,org.apache.spark.sql.comet.DeltaPlanDataInjectorSuite

# Saved only on main: an entry written from a pull request or a
# `gh-readonly-queue/*` branch cannot be restored by any later run, and
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/iceberg_spark_test_reusable.yml
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@ jobs:
needs: iceberg-spark
if: ${{ !cancelled() }}
name: iceberg-spark-shard-coverage/iceberg-${{ inputs.iceberg-full }}/spark-${{ inputs.spark-full }}/scala-${{ inputs.scala }}/java-${{ inputs.java }}
runs-on: ubuntu-slim
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v7
- name: Download Iceberg shard inventories
Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/pr_build_linux.yml
Original file line number Diff line number Diff line change
Expand Up @@ -310,6 +310,9 @@ jobs:
restore-keys: |
${{ runner.os }}-java-maven-

- name: Resolve Maven model before strict compilation
run: dev/ci/prepare-maven-model.sh -Pspark-3.5,delta -Pstrict-warnings

- name: Compile with strict warnings
run: ./mvnw -B test-compile -Pspark-3.5,delta -Pstrict-warnings -DskipTests

Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/pr_title_check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ on:

jobs:
check-pr-title:
runs-on: ubuntu-slim
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v7
- name: Check PR title
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/pyarrow_udf_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ jobs:
native_python_udf: true
- name: Spark 4.2
maven_profiles: "-Pspark-4.2"
pyspark: "4.2.0"
pyspark: "4.2.1"
native_python_udf: true
container:
# Pinned to the Debian 12 (bookworm) base so the system `python3` is 3.11. The default
Expand Down
Loading
Loading