Skip to content

ci: fork runs never get a Maven cache, so Maven Central throttles them (429) #7702

Open

Description

@dwsmith1983

Describe the bug

Since the pull request tier moved to contributors' forks (#7311), most fork runs fail before building anything because Maven cannot fetch dependencies from Maven Central. The usual first failure is the root parent POM:

[FATAL] Non-resolvable parent POM for org.apache.datafusion:comet-parent-spark${spark.version.short}_${scala.binary.version}:1.3.0-SNAPSHOT:
The following artifacts could not be resolved: org.apache:apache:pom:23 (absent): Could not transfer artifact org.apache:apache:pom:23
from/to central (https://repo.maven.apache.org/maven2): status code: 429, reason phrase: Too Many Requests

Some runs show Could not find artifact org.apache:apache:pom:23 in central instead. The Lint Scala job fails the same way inside coursier (CantDownloadModule: Error downloading org.scala-sbt:sbt:latest.stable).

It isn't limited to one fork. Over the past week the same errors show up in fork runs from at least six contributors, and they hit lint, rust-test, the Spark SQL and Iceberg shards, Celeborn and TPC-DS jobs more or less at random. In ten failed runs on my fork today, every failed job was a download failure and none had a test failure. Runs on apache's own runners (main, merge queue) are unaffected.

The cause looks like cache scoping. Maven repository caches are restored everywhere but saved only on push to main (see "Large caches are written on main only" in .github/workflows/README.md). A fork's main never runs that job, so the fork never has a Linux-java-maven-* entry. Every Maven job in every fork run logs Cache not found for input keys: Linux-java-maven-... and downloads the full dependency set from Central, roughly 30 cold jobs per push. The four retries in maven-bootstrap don't survive throttling that lasts for hours.

Steps to reproduce

Push a branch to a fork and look at the fork run of Comet CI (fork run), or at any CI (fork) check that failed this week.

Expected behavior

Fork runs restore a warm Maven repository and touch Central only when pom.xml changes.

Additional context

#1507 reported the same 429 in 2025, before fork CI. Some possible directions:

  • On fork runs, let one job per run resolve dependencies and save the Maven repository, and have the other jobs restore it. A fork's caches count against the fork's own budget, so the main-only rule's reason (protecting apache's shared budget) doesn't apply there.
  • Run the cache-refresh-only build on push to a fork's main, so a synced fork has warm entries.
  • Point fork runs at a Central mirror.

Activity

  1. dwsmith1983 commented on Nov 20, 2026

    @dwsmith1983
    ContributorAuthor

    take

  2. added
    bugSomething isn't working
    priority:lowMinor issues, test failures, tooling, cosmetic
    area:ciCI/CD, GitHub Actions, build tooling
    and removed on Nov 20, 2026
  3. dwsmith1983 commented on Nov 20, 2026

    @dwsmith1983
    ContributorAuthor

    A separate finding while working on this, which affects upstream rather than forks.

    Upstream main often has no Linux-java-maven-* or Linux-cargo-ci-* cache. Each push to main saves them, but the nightly then writes its Linux-delta-maven-* entries (about 7 GB across four entries today, saved 06:48 to 07:11Z), which takes the repository past the 10 GB cache budget. GitHub evicts the older entries, so until the next push to main, pull request and merge queue runs restore no Maven or cargo cache and download from Maven Central, the same as a fork. Linux-spark-sql-* usually survives, which is why #7706 uses it as a fallback key.

    @andygrove, in case this is worth its own issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:ciCI/CD, GitHub Actions, build toolingbugSomething isn't workingpriority:lowMinor issues, test failures, tooling, cosmetic

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions