GH-1283: Optimize number of build compilations in CI by xborder · Pull Request #1284 · apache/arrow-java

xborder · 2026-09-04T22:46:04Z

What's Changed

Build Java artifacts once with JDK 17 and reuse them across the test matrix.
Build C Data JNI artifacts once and reuse them across JDK 17, 21, and 25.
Distribute compiled classes and Maven artifacts through GitHub Actions artifacts.
Run prebuilt tests directly with Surefire to avoid recompilation.
Preserve Vector allocator and memory-core custom test executions.
Add checks to ensure test classes are restored and no unexpected compilation occurs.
Keep pure-Java tests running on Ubuntu, macOS Intel, macOS ARM, and Windows.
Reduce duplicate compilation while retaining the existing platform and JDK test coverage.
Ran a few tests, managed to reduce the total runner time by ~50%

Closes #1283.

This PR was assisted by AI

jbonofre · 2026-09-06T05:08:39Z

Thanks! I gonna take a look.

xborder · 2026-09-22T14:45:11Z

@jbonofre can you also take a look at this one?

xborder · 2026-09-28T13:26:53Z

@kou since you took a look at my last 2 CI changes, can you also take a look at this one?

kou · 2026-09-28T21:46:30Z

+          path: java-build.tgz
+          retention-days: 1
+      - name: Exclude reactor artifacts from Maven dependency cache
+        run: rm -rf .docker/maven-cache/repository/org/apache/arrow


Why do we need to remove them? We can't use them to avoid rebuilding Arrow Java, right?

It is removed so the cache doesn’t carry old Arrow binaries into future runs. The binaries are already in the build artifacts used by every test job

kou · 2026-09-28T21:49:10Z

          - arch: AArch64
            jdk: 17
-            macos: latest


It seems that we need os: macos-XXX here.

not sure I understood. Isn't that what was done here?

kou · 2026-09-28T21:54:50Z

+      - name: Cache Docker Volumes
+        if: ${{ matrix.compose_service }}
+        uses: actions/cache@v6
+        with:
+          path: .docker/maven-cache
+          key: java-build-${{ env.BUILD_JDK }}-${{ env.MAVEN }}-${{ hashFiles('compose.yaml', '**/pom.xml') }}
+          restore-keys: java-build-${{ env.BUILD_JDK }}-${{ env.MAVEN }}-


Why doe we need this?
download-artifact isn't enough?

this avoids downloading external dependencies again in each job. download-artifacts only contains the compiled Arrow output and installed org/apache/arrow artifacts

jbonofre · 2026-10-04T04:44:19Z

  contents: read

 env:
+  BUILD_JDK: "17"


Building once with JDK 17 means Spotless and Error Prone no longer run in this workflow: both profiles in the root pom.xml are activated with <jdk>[21,)</jdk>. On main the JDK 21 and 25 jobs run build.sh with those JDKs, so formatting and Error Prone violations fail the PR. With this change they would pass.

It also means nothing compiles the sources with JDK 21 or 25 any more: the jdk: [17, 21, 25] matrix only runs classes compiled by 17, so a change that breaks under a newer javac would go unnoticed.

Could you add a compile-only job (no tests) on JDK 25, or on 21 and 25? That brings both checks back and should only cost a few minutes of the time saved.

jbonofre · 2026-10-04T04:47:52Z

          DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
-        run: ci/scripts/build.sh . build jni
-      - name: Test
+        run: ci/scripts/test.sh . build jni


On main the Windows job runs ci/scripts/build.sh before test.sh. Here it only runs Surefire against classes compiled on Linux, and as far as I can see that leaves no Maven build on Windows in any workflow (jni-windows in rc.yml only runs jni_windows_build.sh, which is CMake).

A change that breaks the build on Windows (codegen paths, a plugin without a Windows binary, line-ending-sensitive checks) would no longer be caught. I would keep the full build for the Windows job. If we decide that is acceptable to drop, please say so in the PR description, which currently states that the existing platform coverage is retained.

I'm more in favor to keep Windows job.

jbonofre · 2026-10-04T04:50:01Z

+        with:
+          name: java-build
+          path: java-build.tgz
+          retention-days: 1


With one day of retention, "Re-run failed jobs" stops working a day after the run: build-java succeeded so it is not re-run, and the test jobs then fail on download-artifact. The only way out is re-running everything, which defeats the purpose.

Could you raise this, for example to 7 days? Same for the cdata-build at line 208.

jbonofre · 2026-10-04T04:52:46Z

+    <<: *cdata-prebuilt
+    environment:
+      ARROW_JAVA_CDATA: "ON"
+      ARROW_JAVA_TEST_BASE: "OFF"


With this Conda JNI jobs only run the arrow-c-data tests. On main, conda-jni-cdata runs the full mvn test first and then the c module, so vector, memory, flight and the rest are also exercised on the conda-forge OpenJDK 17/21/25.

Is dropping that intentional? I'm fine with it as deduplication, but then please mention it in PR description, since it says coverage is retained.

This comment has been minimized.

Sign in to view

xborder marked this pull request as ready for review September 5, 2026 04:45

xborder requested review from jbonofre, kou, laurentgo, lidavidm and wgtmac as code owners September 5, 2026 04:45

kou added the chore PRs that make misc changes. label Sep 6, 2026

github-actions Bot added this to the 20.0.0 milestone Sep 6, 2026

kou reviewed Sep 28, 2026

View reviewed changes

xborder added 15 commits October 1, 2026 18:44

MINOR: Build test artifacts once in CI

513a69c

Simplify build-once test workflow

bcc9111

Restore custom Surefire test coverage

792fda8

Make Vector test passes explicit

99574ff

MINOR: Align build-once workflow with current CI matrix

f71a6f6

Fix stale Arrow artifacts in hosted tests

e9591df

Verify prebuilt test artifacts

0fad043

Simplify prebuilt test validation

5fc4f18

Optimize build-once workflow caches

a775dad

Quote workflow build JDK

72e6787

Fix build JDK workflow context

398329f

Use established Maven cache keys

04b74d5

Use shallow test checkouts

9dd2495

Sync setup-java version with upstream

6d4c6eb

Fix test workflow review comments

5309167

xborder force-pushed the ci-build-once-test-many branch from 18a8b8a to 5309167 Compare October 1, 2026 17:45

xborder requested a review from kou October 1, 2026 21:32

jbonofre requested changes Oct 4, 2026

View reviewed changes

Conversation

xborder commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What's Changed

Uh oh!

This comment has been minimized.

jbonofre commented Sep 6, 2026

Uh oh!

xborder commented Sep 22, 2026

Uh oh!

xborder commented Sep 28, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xborder Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

xborder commented Sep 4, 2026 •

edited

Loading

xborder Oct 1, 2026 •

edited

Loading