Repository navigation
ci: shard the Iceberg extensions test task #6113
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
14199c8
dbf0f31
8eb6dfd
a033c18
b0bc125
589001e
4307a38
b745b19
645e48d
cc9fa7e
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -206,8 +206,11 @@ jobs: | |
|
|
||
| iceberg-spark-extensions: | ||
| needs: build-native | ||
| name: iceberg-spark-extensions/iceberg-${{ inputs.iceberg-full }}/spark-${{ inputs.spark-full }}/scala-${{ inputs.scala }}/java-${{ inputs.java }} | ||
| name: iceberg-spark-extensions/iceberg-${{ inputs.iceberg-full }}/spark-${{ inputs.spark-full }}/scala-${{ inputs.scala }}/java-${{ inputs.java }}/shard-${{ matrix.shard }} | ||
| runs-on: ubuntu-24.04 | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: ${{ fromJSON(needs.build-native.outputs.shard-matrix) }} | ||
| container: | ||
| image: amd64/rust | ||
| env: | ||
|
|
@@ -238,21 +241,52 @@ jobs: | |
| ENABLE_COMET=true ENABLE_COMET_ONHEAP=true COMET_ICEBERG_DATA_FILE_CONCURRENCY_LIMIT=1 COMET_ICEBERG_WRITE_REPORT_DIR="$PWD/build/comet-iceberg-writes" \ | ||
| ./gradlew -DsparkVersions=${{ inputs.spark-short }} -DscalaVersion=${{ inputs.scala }} -DflinkVersions= -DkafkaVersions= \ | ||
| :iceberg-spark:iceberg-spark-extensions-${{ inputs.spark-short }}_${{ inputs.scala }}:test \ | ||
| --init-script ../dev/ci/iceberg-test-shards.gradle \ | ||
| -PcometShardTask=:iceberg-spark:iceberg-spark-extensions-${{ inputs.spark-short }}_${{ inputs.scala }}:test \ | ||
| -PcometShardIndex=${{ matrix.shard }} -PcometShardCount=${{ needs.build-native.outputs.shard-count }} \ | ||
| -Pquick=true -x javadoc | ||
| - name: Summarize Iceberg writes | ||
| if: ${{ !cancelled() }} | ||
| run: | | ||
| python3 dev/ci/summarize-iceberg-writes.py --title "iceberg-spark-extensions" \ | ||
| python3 dev/ci/summarize-iceberg-writes.py --title "iceberg-spark-extensions shard ${{ matrix.shard }}" \ | ||
| apache-iceberg/build/comet-iceberg-writes | ||
| - name: Upload Iceberg write report | ||
| - name: Upload Iceberg shard inventory and test reports | ||
| if: ${{ !cancelled() }} | ||
| # iceberg-spark-extensions-shard-coverage downloads the inventory, so a | ||
| # flaky upload here fails that job too. Retry rather than fail on 403. | ||
| uses: ./.github/actions/upload-artifact-retry | ||
| with: | ||
| name: iceberg-spark-extensions-writes-${{ inputs.iceberg-full }}-spark-${{ inputs.spark-full }}-scala-${{ inputs.scala }}-jdk${{ inputs.java }}-attempt-${{ github.run_attempt }} | ||
| path: apache-iceberg/build/comet-iceberg-writes/*.jsonl | ||
| if-no-files-found: ignore | ||
| name: iceberg-spark-extensions-${{ inputs.iceberg-full }}-spark-${{ inputs.spark-full }}-scala-${{ inputs.scala }}-jdk${{ inputs.java }}-shard-${{ matrix.shard }}-attempt-${{ github.run_attempt }} | ||
| path: | | ||
| apache-iceberg/**/build/comet-shards/*.json | ||
| apache-iceberg/**/build/test-results/test/*.xml | ||
| apache-iceberg/build/comet-iceberg-writes/*.jsonl | ||
| retention-days: 7 | ||
|
|
||
| iceberg-spark-extensions-shard-coverage: | ||
| needs: iceberg-spark-extensions | ||
| if: ${{ !cancelled() }} | ||
| name: iceberg-spark-extensions-shard-coverage/iceberg-${{ inputs.iceberg-full }}/spark-${{ inputs.spark-full }}/scala-${{ inputs.scala }}/java-${{ inputs.java }} | ||
| runs-on: ubuntu-slim | ||
|
Comment on lines
+266
to
+270
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. After the rebase this new job also needs a |
||
| steps: | ||
| - uses: actions/checkout@v7 | ||
| - name: Download Iceberg extensions shard inventories | ||
| uses: ./.github/actions/download-artifact-retry | ||
| with: | ||
| # Download all attempts: a failed-job rerun retains earlier successful | ||
| # shards. The checker selects the latest inventory for each index. | ||
| pattern: iceberg-spark-extensions-${{ inputs.iceberg-full }}-spark-${{ inputs.spark-full }}-scala-${{ inputs.scala }}-jdk${{ inputs.java }}-shard-*-attempt-* | ||
| path: iceberg-shard-reports | ||
| - name: Verify complete, disjoint Iceberg extensions candidate coverage | ||
| run: | | ||
| python3 dev/ci/check-iceberg-shards.py --manifests iceberg-shard-reports \ | ||
| --task :iceberg-spark:iceberg-spark-extensions-${{ inputs.spark-short }}_${{ inputs.scala }}:test | ||
| - name: Summarize Iceberg writes across shards | ||
| if: ${{ !cancelled() }} | ||
| run: | | ||
| python3 dev/ci/summarize-iceberg-writes.py --title "iceberg-spark-extensions, all shards" \ | ||
| iceberg-shard-reports | ||
|
|
||
| iceberg-spark-runtime: | ||
| needs: build-native | ||
| name: iceberg-spark-runtime/iceberg-${{ inputs.iceberg-full }}/spark-${{ inputs.spark-full }}/scala-${{ inputs.scala }}/java-${{ inputs.java }} | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[P1] Use an extensions-specific partition before enabling this init script.
CometIcebergTestShard.owner()reserves shard 1 exclusively forTestStructuredStreamingRead*and assigns everything else to shards 2–4. None of the configured extensions source sets contains that family. Consequently, shard 1 becomesNO-SOURCE, Gradle skips thedoFirstinventory writer, and the new coverage job fails because inventory 1 is missing, even when every extensions test passes. This makes the Iceberg workflow fail and blocks merge-queue runs that require it. Preserve the core streaming special case while distributing extensions across all four shards, and add a fixture without streaming tests.Evidence: Checked upstream Iceberg 1.8.1/Spark 3.4, 1.9.1/Spark 3.5, 1.10.0/Spark 3.5, and 1.11.0/Spark 4.1 source trees and Gradle configuration:
TestStructuredStreamingRead3belongs to the core module, not extensions. Reproduced with Gradle 8.14.4/JDK 17 using isolated Java test projects containing passingTestExtendedParser,TestAddFilesProcedure, andTestAlterTableSchematests and the unchanged repository init script with-PcometShardTask=:test -PcometShardIndex=1..4 -PcometShardCount=4. Baseline executed three tests. Shard 1 reported:test NO-SOURCEand wrote no manifest. Shards 2–4 each passed one test and wrote their inventories. The repository coverage checker exited 1 withExpected shard indices [1, 2, 3, 4], found [2, 3, 4]. Reproduction and logs:/tmp/comet-6113-repro-n141sa7s/run-isolated-repro.pyand itsisolated/directory.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Addressed in
589001e: extensions distribute across all four shards, while the core streaming reservation remains intact. The regression fixtures cover extensions without streaming tests, including small and empty inventories.Rechecked the current head: all 15 tests in
dev/ci/test-iceberg-shards.pyand the CI configuration guard passed. The current Comet CI and CodeQL runs are still awaiting approval. Could a maintainer approve them so the hosted jobs can verify this head?