Skip to content

GH-1283: Optimize number of build compilations in CI #1284

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
xborder wants to merge 15 commits into
base: main
Choose a base branch
from
+299 −59
Open

GH-1283: Optimize number of build compilations in CI #1284

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
513a69c
MINOR: Build test artifacts once in CI
xborder Jul 28, 2026
bcc9111
Simplify build-once test workflow
xborder Jul 29, 2026
792fda8
Restore custom Surefire test coverage
xborder Jul 31, 2026
99574ff
Make Vector test passes explicit
xborder Jul 31, 2026
f71a6f6
MINOR: Align build-once workflow with current CI matrix
xborder Sep 2, 2026
e9591df
Fix stale Arrow artifacts in hosted tests
xborder Sep 2, 2026
0fad043
Verify prebuilt test artifacts
xborder Sep 2, 2026
5fc4f18
Simplify prebuilt test validation
xborder Sep 2, 2026
a775dad
Optimize build-once workflow caches
xborder Sep 3, 2026
72e6787
Quote workflow build JDK
xborder Sep 3, 2026
398329f
Fix build JDK workflow context
xborder Sep 3, 2026
04b74d5
Use established Maven cache keys
xborder Sep 3, 2026
9dd2495
Use shallow test checkouts
xborder Sep 3, 2026
6d4c6eb
Sync setup-java version with upstream
xborder Sep 4, 2026
5309167
Fix test workflow review comments
xborder Sep 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
208 changes: 158 additions & 50 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,28 +34,16 @@ permissions:
contents: read

env:
BUILD_JDK: "17"

@jbonofre jbonofre Oct 4, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Building once with JDK 17 means Spotless and Error Prone no longer run in this workflow: both profiles in the root pom.xml are activated with <jdk>[21,)</jdk>. On main the JDK 21 and 25 jobs run build.sh with those JDKs, so formatting and Error Prone violations fail the PR. With this change they would pass.

It also means nothing compiles the sources with JDK 21 or 25 any more: the jdk: [17, 21, 25] matrix only runs classes compiled by 17, so a change that breaks under a newer javac would go unnoticed.

Could you add a compile-only job (no tests) on JDK 25, or on 21 and 25? That brings both checks back and should only cost a few minutes of the time saved.

DOCKER_VOLUME_PREFIX: ".docker/"
MAVEN: 3.9.16

jobs:
ubuntu:
name: AMD64 ${{ matrix.name }} JDK ${{ matrix.jdk }} Maven ${{ matrix.maven }}
build-java:
name: Build Java artifacts
runs-on: ubuntu-latest
if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
jdk: [17, 21, 25]
maven: [3.9.16]
image: [ubuntu, conda-jni-cdata]
include:
- image: ubuntu
name: "Ubuntu"
- image: conda-jni-cdata
name: "Conda JNI"
env:
JDK: ${{ matrix.jdk }}
MAVEN: ${{ matrix.maven }}
steps:
- name: Checkout Arrow
uses: actions/checkout@v7
Expand All @@ -65,82 +53,202 @@ jobs:
- name: Cache Docker Volumes
uses: actions/cache@v6
with:
path: .docker
key: maven-${{ matrix.jdk }}-${{ matrix.maven }}-${{ hashFiles('compose.yaml', '**/pom.xml') }}
restore-keys: maven-${{ matrix.jdk }}-${{ matrix.maven }}-
- name: Execute Docker Build
path: .docker/maven-cache
key: java-build-${{ env.BUILD_JDK }}-${{ env.MAVEN }}-${{ hashFiles('compose.yaml', '**/pom.xml') }}
restore-keys: java-build-${{ env.BUILD_JDK }}-${{ env.MAVEN }}-
- name: Build without tests
env:
# Enables build caching, but not strictly required
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
JDK: ${{ env.BUILD_JDK }}
run: |
docker compose run \
--rm \
-e CI=true \
-e "DEVELOCITY_ACCESS_KEY=$DEVELOCITY_ACCESS_KEY" \
${{ matrix.image }}
ubuntu-artifacts
sudo chown -R "$(id -u):$(id -g)" .docker
- name: Pack reusable artifacts
run: |
tar -czf java-build.tgz \
.docker/java-build \
.docker/maven-cache/repository/org/apache/arrow
- name: Upload reusable artifacts
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: java-build
path: java-build.tgz
retention-days: 1

@jbonofre jbonofre Oct 4, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With one day of retention, "Re-run failed jobs" stops working a day after the run: build-java succeeded so it is not re-run, and the test jobs then fail on download-artifact. The only way out is re-running everything, which defeats the purpose.

Could you raise this, for example to 7 days? Same for the cdata-build at line 208.

- name: Exclude reactor artifacts from Maven dependency cache
run: rm -rf .docker/maven-cache/repository/org/apache/arrow

@kou kou Sep 28, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to remove them? We can't use them to avoid rebuilding Arrow Java, right?

@xborder xborder Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is removed so the cache doesn’t carry old Arrow binaries into future runs. The binaries are already in the build artifacts used by every test job


macos:
name: ${{ matrix.arch }} macOS ${{ matrix.macos }} Java JDK ${{ matrix.jdk }}
runs-on: macos-${{ matrix.macos }}
test-java:
name: ${{ matrix.name || format('AMD64 Ubuntu JDK {0} Maven 3.9.16', matrix.jdk) }}
needs: build-java
runs-on: ${{ matrix.os }}
if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
jdk: [17, 21, 25]
os: [ubuntu-latest]
include:
- arch: AArch64
- os: ubuntu-latest
compose_service: ubuntu-test-prebuilt
- name: AArch64 macOS latest Java JDK 17
os: macos-latest
jdk: 17
macos: latest
compose_service: ''
- name: AMD64 Windows Server 2022 Java JDK 17
os: windows-latest
jdk: 17
compose_service: ''
env:
JDK: ${{ matrix.jdk }}
steps:
- name: Checkout Arrow
uses: actions/checkout@v7
with:
fetch-depth: 0
submodules: recursive
- name: Set up Java
if: ${{ !matrix.compose_service }}
Comment thread
xborder marked this conversation as resolved.
uses: actions/setup-java@v6
with:
distribution: 'temurin'
java-version: ${{ matrix.jdk }}
cache: 'maven'
- name: Build
- name: Cache Docker Volumes
if: ${{ matrix.compose_service }}
uses: actions/cache@v6
with:
path: .docker/maven-cache
key: java-build-${{ env.BUILD_JDK }}-${{ env.MAVEN }}-${{ hashFiles('compose.yaml', '**/pom.xml') }}
restore-keys: java-build-${{ env.BUILD_JDK }}-${{ env.MAVEN }}-
Comment on lines +120 to +126

@kou kou Sep 28, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why doe we need this?
download-artifact isn't enough?

@xborder xborder Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this avoids downloading external dependencies again in each job. download-artifacts only contains the compiled Arrow output and installed org/apache/arrow artifacts

- name: Download reusable artifacts
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: java-build
- name: Restore reusable artifacts
shell: bash
run: |
rm -rf \
.docker/java-build \
.docker/maven-cache/repository/org/apache/arrow
tar -xzf java-build.tgz
- name: Restore reusable artifacts for hosted runner
if: ${{ !matrix.compose_service }}
shell: bash
run: |
cp -a .docker/java-build/build build
rm -rf "${HOME}/.m2/repository/org/apache/arrow"
mkdir -p "${HOME}/.m2/repository/org/apache"
cp -a .docker/maven-cache/repository/org/apache/arrow "${HOME}/.m2/repository/org/apache/"
- name: Test prebuilt artifacts on macOS/Windows
if: ${{ !matrix.compose_service }}
shell: bash
env:
ARROW_JAVA_TEST_PREBUILT: "ON"
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
run: ci/scripts/build.sh . build jni
- name: Test
run: ci/scripts/test.sh . build jni

@jbonofre jbonofre Oct 4, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On main the Windows job runs ci/scripts/build.sh before test.sh. Here it only runs Surefire against classes compiled on Linux, and as far as I can see that leaves no Maven build on Windows in any workflow (jni-windows in rc.yml only runs jni_windows_build.sh, which is CMake).

A change that breaks the build on Windows (codegen paths, a plugin without a Windows binary, line-ending-sensitive checks) would no longer be caught. I would keep the full build for the Windows job. If we decide that is acceptable to drop, please say so in the PR description, which currently states that the existing platform coverage is retained.

I'm more in favor to keep Windows job.

- name: Test prebuilt artifacts on Ubuntu Docker
if: ${{ matrix.compose_service }}
shell: bash
env:
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
run: ci/scripts/test.sh . build jni
run: |
docker compose run \
--rm \
-e CI=true \
-e "DEVELOCITY_ACCESS_KEY=$DEVELOCITY_ACCESS_KEY" \
${{ matrix.compose_service }}
- name: Exclude reactor artifacts from hosted Maven dependency cache
if: ${{ !matrix.compose_service && always() }}
shell: bash
run: rm -rf "${HOME}/.m2/repository/org/apache/arrow"

windows:
name: AMD64 Windows Server 2022 Java JDK ${{ matrix.jdk }}
runs-on: windows-latest
build-cdata:
name: Build C Data artifacts
runs-on: ubuntu-latest
if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
jdk: [17]
steps:
- name: Checkout Arrow
uses: actions/checkout@v7
with:
fetch-depth: 0
submodules: recursive
- name: Set up Java
uses: actions/setup-java@v6
- name: Cache Docker Volumes
uses: actions/cache@v6
with:
java-version: ${{ matrix.jdk }}
distribution: 'temurin'
cache: 'maven'
- name: Build
shell: bash
path: .docker/maven-cache
key: cdata-${{ env.BUILD_JDK }}-${{ env.MAVEN }}-${{ hashFiles('compose.yaml', '**/pom.xml') }}
restore-keys: cdata-${{ env.BUILD_JDK }}-${{ env.MAVEN }}-
- name: Build C Data without tests
env:
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
run: ci/scripts/build.sh . build jni
- name: Test
shell: bash
JDK: ${{ env.BUILD_JDK }}
run: |
docker compose run \
--rm \
-e CI=true \
-e "DEVELOCITY_ACCESS_KEY=$DEVELOCITY_ACCESS_KEY" \
cdata-artifacts
sudo chown -R "$(id -u):$(id -g)" .docker
- name: Pack reusable C Data artifacts
run: |
tar -czf cdata-build.tgz \
.docker/cdata-build \
.docker/cdata-jni-dist \
.docker/maven-cache/repository/org/apache/arrow
- name: Upload reusable C Data artifacts
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: cdata-build
path: cdata-build.tgz
retention-days: 1
- name: Exclude reactor artifacts from Maven dependency cache
run: rm -rf .docker/maven-cache/repository/org/apache/arrow

test-cdata:
name: AMD64 Conda JNI JDK ${{ matrix.jdk }} Maven 3.9.16
needs: build-cdata
runs-on: ubuntu-latest
if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
jdk: [17, 21, 25]
env:
JDK: ${{ matrix.jdk }}
steps:
- name: Checkout Arrow
uses: actions/checkout@v7
with:
submodules: recursive
- name: Cache Docker Volumes
uses: actions/cache@v6
with:
path: .docker/maven-cache
key: cdata-${{ env.BUILD_JDK }}-${{ env.MAVEN }}-${{ hashFiles('compose.yaml', '**/pom.xml') }}
restore-keys: cdata-${{ env.BUILD_JDK }}-${{ env.MAVEN }}-
- name: Download reusable C Data artifacts
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: cdata-build
- name: Restore reusable C Data artifacts
run: |
rm -rf \
.docker/cdata-build \
.docker/cdata-jni-dist \
.docker/maven-cache/repository/org/apache/arrow
tar -xzf cdata-build.tgz
- name: Test C Data without compiler lifecycle
env:
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
run: ci/scripts/test.sh . build jni
run: |
docker compose run \
--rm \
-e CI=true \
-e "DEVELOCITY_ACCESS_KEY=$DEVELOCITY_ACCESS_KEY" \
cdata-test-prebuilt
2 changes: 1 addition & 1 deletion ci/scripts/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ cp -r "${source_dir}/dev" "${build_dir}"
# crawl back up to the top. GNU realpath has --relative-to but this does not
# work on macOS

poms=$(find "${source_dir}" -not \( -path "${source_dir}"/build -prune \) -type f -name pom.xml)
poms=$(find "${source_dir}" \( -path "${source_dir}"/build -o -path "${source_dir}"/.docker \) -prune -o -type f -name pom.xml -print)
if [[ "$OSTYPE" == "darwin"* ]]; then
poms=$(echo "$poms" | xargs -n1 python -c "import sys; import os.path; print(os.path.relpath(sys.argv[1], '${source_dir}'))")
else
Expand Down
79 changes: 71 additions & 8 deletions ci/scripts/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
# specific language governing permissions and limitations
# under the License.

set -ex
set -exo pipefail

if [[ "${ARROW_JAVA_TEST:-ON}" != "ON" ]]; then
exit
Expand All @@ -31,14 +31,72 @@ if [ -d "${java_jni_dist_dir}" ]; then
java_jni_dist_dir="$(cd "${java_jni_dist_dir}" && pwd)"
fi

mvn="mvn -B -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn"
# Use `2 * ncores` threads
mvn="${mvn} -T 2C"
mvn="${mvn} -Denforcer.skip=true"
mvn=(
mvn
-B
-Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn
-T
2C
-Denforcer.skip=true
)

run_tests() {
if [[ "${ARROW_JAVA_TEST_PREBUILT:-OFF}" = "ON" ]]; then
run_prebuilt_tests "${@}" surefire:test
else
"${@}" test
fi
}

run_prebuilt_tests() {
local log
log=$(mktemp)

"${@}" -Ddevelocity.cache.local.enabled=false | tee "${log}"

if grep -E "Compiling [0-9]+ source files?" "${log}"; then
echo "Unexpected compilation occurred while running prebuilt tests."
exit 1
fi

if ! grep -q "Tests run:" "${log}"; then
echo "No surefire test summary found; tests may have been skipped."
exit 1
fi

rm -f "${log}"
}

pushd "${build_dir}"

${mvn} -Darrow.test.dataRoot="${source_dir}/testing/data" test
if [[ "${ARROW_JAVA_TEST_BASE:-ON}" = "ON" ]]; then
if [[ "${ARROW_JAVA_TEST_PREBUILT:-OFF}" = "ON" ]]; then
run_prebuilt_tests \
"${mvn[@]}" \
-Darrow.test.dataRoot="${source_dir}/testing/data" \
-pl "!vector" \
surefire:test
# Direct Surefire skips Vector's lifecycle-bound allocator test passes.
run_prebuilt_tests \
"${mvn[@]}" \
-Darrow.test.dataRoot="${source_dir}/testing/data" \
-pl vector \
org.apache.maven.plugins:maven-surefire-plugin:test@default-test
run_prebuilt_tests \
"${mvn[@]}" \
-Darrow.test.dataRoot="${source_dir}/testing/data" \
-pl vector \
org.apache.maven.plugins:maven-surefire-plugin:test@run-unsafe
run_prebuilt_tests \
"${mvn[@]}" \
-pl memory/memory-core \
org.apache.maven.plugins:maven-surefire-plugin:test@opens-tests
else
run_tests \
"${mvn[@]}" \
-Darrow.test.dataRoot="${source_dir}/testing/data"
fi
fi

projects=()
if [ "${ARROW_JAVA_JNI}" = "ON" ]; then
Expand All @@ -47,7 +105,8 @@ if [ "${ARROW_JAVA_JNI}" = "ON" ]; then
projects+=(gandiva)
fi
if [ "${#projects[@]}" -gt 0 ]; then
${mvn} test \
run_tests \
"${mvn[@]}" \
-Parrow-jni \
-pl "$(
IFS=,
Expand All @@ -57,7 +116,11 @@ if [ "${#projects[@]}" -gt 0 ]; then
fi

if [ "${ARROW_JAVA_CDATA}" = "ON" ]; then
${mvn} test -Parrow-c-data -pl c -Darrow.c.jni.dist.dir="${java_jni_dist_dir}"
run_tests \
"${mvn[@]}" \
-Parrow-c-data \
-pl c \
-Darrow.c.jni.dist.dir="${java_jni_dist_dir}"
fi

popd
Loading
Loading