Skip to content

Fix the two causes of the intermittent Neo4j functional test failures - #16436

Merged
jdaugherty merged 1 commit into
8.0.xfrom
fix/neo4j-ci-flakes
Sep 30, 2026
Merged

jdaugherty merged 1 commit into
8.0.xfrom
fix/neo4j-ci-flakes

Conversation

@jdaugherty

Copy link
Copy Markdown
Contributor

Description

The Neo4j functional test jobs fail intermittently on 8.0.x, on whichever JDK of the matrix happens to be unlucky (#16435 is a recent example). A survey of every gradle.yml run since 2026-09-16 showed that all of the Neo4j failures not caused by the PR under test fall into two shapes.

1. Dependency resolution dies on repo.grails.org before any test runs

Could not GET 'https://repo.grails.org/grails/restricted/de/flapdoodle/embed/de.flapdoodle.embed.mongo/4.33.0/de.flapdoodle.embed.mongo-4.33.0.pom'
  > Remote host terminated the handshake

Seen on #16435 (Java 25, 2026-09-30) and, on maven-resolver-api, on the 2026-09-23 run for #16385. Three things combine here:

  • GrailsRepoSettingsPlugin declared repo.grails.org/grails/restricted as the only release repository for project dependencies, so every POM and jar of the build went through that one host. Gradle disables a repository for the rest of the build on a transport failure and fails resolution; it never falls through to the next repository.
  • The Neo4j, Hibernate 7, Redis, Spring Security and validate jobs used setup-gradle with cache-provider: basic. That provider keys its entry as setup-java-<os>-<arch>-gradle-<hash of **/*.gradle*>, has no restore keys, and shares the one entry between every job that uses it. Whichever job finishes first on an 8.0.x push writes it; that is the validate job, so the entry is ~290-350 MB and holds none of the GORM test dependencies.
  • Every Neo4j run therefore re-downloaded the rest of the dependency set through repo.grails.org. On the 2026-09-24 push run the Neo4j, Hibernate 7 and Redis jobs each spent about 3.5 minutes resolving, against about 1 minute for the Mongo job that uses the branch-keyed actions/cache step from 6daf2d1. A PR that touches any Gradle file started fully cold.

2. initializationError ending in org.neo4j.helpers.PortBindException

MultiTenancySpec lost on 2026-09-23 (#16385) and HasManyInheritanceSpec on 2026-09-27 (#16415). Both specs configured their embedded Neo4j with bolt://localhost:7687, and Neo4jConnectionSourceFactory binds that literal port when a URL is given. grails-data-neo4j-core:test runs with maxParallelForks = 4 and forkEvery = 10 on CI, so whenever the two specs are dispatched to different forks at the same time the second one cannot bind. The forks and the pinned port arrived together in 413d19f (#15972). The TCK manager already lets the harness pick a free port.

Changes

  • .github/workflows/gradle.yml: the eight jobs on cache-provider: basic now use the same branch-keyed actions/cache step as the rest of the workflow (gradle-deps-<os>-<branch>-<hash of dependencies.gradle and the wrapper>, currently ~1.1 GB and complete), with cache-disabled: true on setup-gradle. Per-job entries were considered and rejected: the repository sits at 10.5 GB of its 10 GB Actions cache limit, and every test family's working set is roughly the same ~1 GB, so six per-job entries per active branch would evict everything and make every job cold.
  • GrailsRepoSettingsPlugin: Maven Central is now the first release repository for project dependencies; repo.grails.org/grails/restricted stays behind it as the fallback for anything Central does not host. pluginManagement already had Central first. A new GrailsRepoSettingsPluginSpec (TestKit) pins the repository order for both blocks, FAIL_ON_PROJECT_REPOS, and the GRAILS_INCLUDE_MAVEN_LOCAL behaviour.
  • MultiTenancySpec / HasManyInheritanceSpec: drop the pinned URL so the embedded server binds a random free port, the same way the TCK manager does. SingleTenancySpec and MultipleConnectionsSpec also mention 7687 but use the default remote connection type, so they never bind it and are unchanged.

Not in this PR: the other workflows that still use cache-provider: basic (codestyle, coverage, rat, sitemesh2, benchmark, codeanalysis, release) keep sharing the small setup-java-* entry. They no longer influence gradle.yml and can move to the branch-keyed cache separately. The 9.0.x Neo4j job timing out after 90 minutes on 2026-09-26 is a different problem on that branch.

Verification

  • build-logic: ./gradlew :build-logic:test --tests org.apache.grails.buildsrc.GrailsRepoSettingsPluginSpec (4 tests, 0 failures)
  • root: ./gradlew :grails-data-neo4j-core:test :grails-data-neo4j-core:codeStyle validateRepositoryConventions (596 tests, 98 skipped, 0 failures; Checkstyle and CodeNarc clean; no repository convention violations)

Contributor Checklist

Issue and Scope

  • This PR is linked to an existing issue that has been acknowledged or approved by the project team. If no approved issue exists, please give background on why this change is necessary. Tickets are preferred for release change log history.
  • This PR addresses the complete scope of the linked issue. Partial implementations or unfinished work should not be submitted for review.
  • This PR contains a single, focused change. Unrelated changes should be submitted as separate pull requests.
  • This PR targets the correct branch for the type of change.

Code Quality

  • I have added or updated tests that cover the changes introduced in this PR.
  • I have verified that all existing tests pass by running ./gradlew build --rerun-tasks.
  • My code follows the project's code style guidelines. I have run ./gradlew codeStyle and resolved any violations.
  • This PR does not include mass reformatting, style-only changes, or large-scale refactoring.

Licensing and Attribution

  • All contributed code is provided under the Apache License 2.0, and new source files include the appropriate Apache license header.
  • I have the necessary rights to submit this contribution and confirm it is my own original work.

Documentation

  • This PR introduces no user-facing changes.
  • The PR description clearly explains what was changed and why.

The Neo4j CI jobs failed on random JDKs of the matrix for two unrelated
reasons, found by surveying every gradle.yml run since 2026-09-16:

1. Dependency resolution died on repo.grails.org ("Remote host terminated
   the handshake") before any test ran. GrailsRepoSettingsPlugin declared
   repo.grails.org/grails/restricted as the only release repository, and
   Gradle never falls through to another repository on a transport
   failure. The jobs using setup-gradle's basic cache provider all shared
   one ~300 MB entry written by the fast validate job, so every Neo4j run
   re-downloaded the GORM dependency set through that host.

2. MultiTenancySpec and HasManyInheritanceSpec both started an embedded
   Neo4j on the pinned port 7687. grails-data-neo4j-core runs its tests
   across parallel forks, so whenever the two specs ran concurrently the
   second one failed with PortBindException.

Maven Central is now the first release repository, with repo.grails.org
kept as the fallback for anything Central does not host; a TestKit spec
pins the repository order. The eight basic-provider CI jobs use the same
branch-keyed dependency jar cache as the other jobs instead of per-job
entries, because the repository is already at its 10 GB cache limit and
every test family needs roughly the same ~1 GB working set. The two specs
drop the pinned URL so the embedded server binds a random free port like
the TCK manager already does.
@jdaugherty jdaugherty added this to the grails:8.0.0 milestone Sep 30, 2026
@codecov

codecov Bot commented Sep 30, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 58.6142%. Comparing base (d521a74) to head (d03fbba).
⚠️ Report is 22 commits behind head on 8.0.x.

Additional details and impacted files

Impacted file tree graph

@@                 Coverage Diff                 @@
##                8.0.x     #16436         +/-   ##
===================================================
+ Coverage     32.7604%   58.6142%   +25.8538%     
- Complexity        563      24309      +23746     
===================================================
  Files              84       2174       +2090     
  Lines            4945     107346     +102401     
  Branches          865      19566      +18701     
===================================================
+ Hits             1620      62920      +61300     
- Misses           3065      35571      +32506     
- Partials          260       8855       +8595     

see 2090 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@testlens-app

testlens-app Bot commented Sep 30, 2026

Copy link
Copy Markdown

✅ All tests passed ✅

🏷️ Commit: d03fbba
▶️ Tests: 89863 executed
⚪️ Checks: 84/84 completed


Learn more about TestLens at testlens.app/docs.

@jdaugherty
jdaugherty merged commit 6f95ce4 into 8.0.x Sep 30, 2026
86 of 87 checks passed
@jdaugherty
jdaugherty deleted the fix/neo4j-ci-flakes branch September 30, 2026 20:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants