Repository navigation
Description
Describe the bug
Since the pull request tier moved to contributors' forks (#7311), most fork runs fail before building anything because Maven cannot fetch dependencies from Maven Central. The usual first failure is the root parent POM:
[FATAL] Non-resolvable parent POM for org.apache.datafusion:comet-parent-spark${spark.version.short}_${scala.binary.version}:1.3.0-SNAPSHOT:
The following artifacts could not be resolved: org.apache:apache:pom:23 (absent): Could not transfer artifact org.apache:apache:pom:23
from/to central (https://repo.maven.apache.org/maven2): status code: 429, reason phrase: Too Many Requests
Some runs show Could not find artifact org.apache:apache:pom:23 in central instead. The Lint Scala job fails the same way inside coursier (CantDownloadModule: Error downloading org.scala-sbt:sbt:latest.stable).
It isn't limited to one fork. Over the past week the same errors show up in fork runs from at least six contributors, and they hit lint, rust-test, the Spark SQL and Iceberg shards, Celeborn and TPC-DS jobs more or less at random. In ten failed runs on my fork today, every failed job was a download failure and none had a test failure. Runs on apache's own runners (main, merge queue) are unaffected.
The cause looks like cache scoping. Maven repository caches are restored everywhere but saved only on push to main (see "Large caches are written on main only" in .github/workflows/README.md). A fork's main never runs that job, so the fork never has a Linux-java-maven-* entry. Every Maven job in every fork run logs Cache not found for input keys: Linux-java-maven-... and downloads the full dependency set from Central, roughly 30 cold jobs per push. The four retries in maven-bootstrap don't survive throttling that lasts for hours.
Steps to reproduce
Push a branch to a fork and look at the fork run of Comet CI (fork run), or at any CI (fork) check that failed this week.
Expected behavior
Fork runs restore a warm Maven repository and touch Central only when pom.xml changes.
Additional context
#1507 reported the same 429 in 2025, before fork CI. Some possible directions:
- On fork runs, let one job per run resolve dependencies and save the Maven repository, and have the other jobs restore it. A fork's caches count against the fork's own budget, so the main-only rule's reason (protecting apache's shared budget) doesn't apply there.
- Run the cache-refresh-only build on push to a fork's main, so a synced fork has warm entries.
- Point fork runs at a Central mirror.
Activity
dwsmith1983 commentedon Nov 20, 2026
take
dwsmith1983 commentedon Nov 20, 2026
A separate finding while working on this, which affects upstream rather than forks.
Upstream
mainoften has noLinux-java-maven-*orLinux-cargo-ci-*cache. Each push tomainsaves them, but the nightly then writes itsLinux-delta-maven-*entries (about 7 GB across four entries today, saved 06:48 to 07:11Z), which takes the repository past the 10 GB cache budget. GitHub evicts the older entries, so until the next push tomain, pull request and merge queue runs restore no Maven or cargo cache and download from Maven Central, the same as a fork.Linux-spark-sql-*usually survives, which is why #7706 uses it as a fallback key.@andygrove, in case this is worth its own issue.