Skip to content

Run Bazel tests in parallel in CI - #3482

Open
wasphin wants to merge 8 commits into
apache:masterfrom
wasphin:optimize-ci-bazel-tests
Open

Run Bazel tests in parallel in CI#3482
wasphin wants to merge 8 commits into
apache:masterfrom
wasphin:optimize-ci-bazel-tests

Conversation

@wasphin

@wasphin wasphin commented Aug 23, 2026

Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue Number: resolve

Problem Summary:

The Bazel unit-test jobs run independent test binaries serially or without an
explicit concurrency limit. This makes the jobs unnecessarily slow, while
unrestricted concurrency exposes tests that rely on timing, fixed ports, or
process-wide networking state to resource contention.

The Bazel disk-cache fallback can also restore an archive produced by a
different compiler or protobuf configuration. Such an archive consumes cache
transfer time and capacity without producing cache hits.

What is changed and the side effects?

Changed:

  • Run the GCC, Clang, and Clang with new protobuf Bazel unit-test jobs with
    --test_output=errors and --local_test_jobs=3.
  • Run timing-sensitive tests and tests that manipulate process-wide networking
    state exclusively, while keeping independent test binaries parallel.
  • Use ephemeral ports in the interceptor, HTTP expect, endpoint, and oversized
    message tests to avoid conflicts with other processes.
  • Set and restore the oversized-message test's maximum body size explicitly.
  • Declare the protobuf JSON test as a large Bazel test so its timeout matches
    its existing runtime under CI contention.
  • Restrict Bazel disk-cache fallback restoration to caches produced by the
    same CI job.

Side effects:

  • Performance effects: reduces Bazel unit-test wall time while limiting local
    test concurrency to avoid overloading GitHub-hosted runners.
  • Breaking backward compatibility: No. Production library behavior is
    unchanged; this updates CI scheduling and test robustness.

Check List:

@wwbmmm

wwbmmm commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

LGTM

@wasphin
wasphin marked this pull request as draft August 24, 2026 15:01
@wasphin
wasphin force-pushed the optimize-ci-bazel-tests branch 7 times, most recently from 2ff532b to d0a517c Compare September 5, 2026 16:21
Run the channel and SSL test binaries exclusively. They rely on strict

timing or process-wide networking state and can fail when competing with

parallel test processes.
Limit fallback restoration to caches produced by the same CI job. The broad

fallback could restore a cache from a different compiler or protobuf setup,

which produced no hits while consuming transfer time and cache capacity.
@wasphin
wasphin force-pushed the optimize-ci-bazel-tests branch from 198adf3 to c719de6 Compare September 6, 2026 16:10
@wasphin
wasphin marked this pull request as ready for review September 6, 2026 16:44
@wasphin
wasphin requested a lite review from Copilot September 6, 2026 16:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The changes are targeted to CI scheduling and test robustness, and the updated test patterns (ephemeral ports, exclusive tags, FlagSaver) are consistent with existing repository practices.

Pull request overview

This PR improves Bazel CI throughput and stability by running unit tests with bounded parallelism and by hardening a set of tests against shared-runner resource conflicts (fixed ports, global flags, and networking state), while also tightening Bazel disk-cache restoration so fallbacks don’t pull incompatible artifacts.

Changes:

  • Update Linux Bazel CI jobs to run tests with --test_output=errors and --local_test_jobs=3, and scope disk-cache fallback restores to the same job.
  • Tag timing-/network-global-state-sensitive Bazel tests as exclusive, and mark brpc_protobuf_json_unittest as large.
  • Make several tests more parallel-safe by using ephemeral ports (Start(0, ...)) and by explicitly scoping gflag mutations with GFLAGS_NAMESPACE::FlagSaver.
File summaries
File Description
test/endpoint_unittest.cpp Uses an ephemeral server port and validates against the actual bound address to avoid port collisions under parallel test runs.
test/BUILD.bazel Adds exclusive tags for contention-sensitive tests and marks a protobuf JSON test as large for CI-timeout alignment.
test/brpc_server_unittest.cpp Avoids fixed ports and scopes FLAGS_max_body_size mutation to the test via FlagSaver.
test/brpc_interceptor_unittest.cpp Switches to ephemeral server port and connects via listen_address() to prevent conflicts.
test/brpc_http_rpc_protocol_unittest.cpp Switches HTTP expect test to an ephemeral port using server.listen_address().
.github/workflows/ci-linux.yml Runs Bazel unit tests with bounded local concurrency and error-only output across GCC/Clang Bazel test jobs.
.github/actions/setup-build-cache/action.yml Restricts Bazel disk-cache restore-keys to per-job caches to avoid restoring incompatible archives.
Review details
  • Files reviewed: 7/7 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants