Skip to content

CI: build Linux binaries for older CPUs/glibc on ubuntu-24.04 - #10344

Merged
ThomasWaldmann merged 1 commit into
borgbackup:masterfrom
ThomasWaldmann:ci-linux-oldglibc-binary
Sep 9, 2026
Merged

CI: build Linux binaries for older CPUs/glibc on ubuntu-24.04#10344
ThomasWaldmann merged 1 commit into
borgbackup:masterfrom
ThomasWaldmann:ci-linux-oldglibc-binary

Conversation

@ThomasWaldmann

@ThomasWaldmann ThomasWaldmann commented Sep 9, 2026

Copy link
Copy Markdown
Member

Fixes #10342.

The Linux binaries from native_tests are built on ubuntu-26.04 and thus need glibc 2.43, and the x86_64 one does not run on CPUs below x86-64-v3 ("Illegal instruction", #10342). This adds the oldglibc_binary job, which builds on ubuntu-24.04 and ubuntu-24.04-arm (glibc 2.39, the oldest GitHub-hosted Ubuntu image, with its baseline x86-64 toolchain and packages) and produces borg-linux-glibc239-{x86_64,arm64}-gh (plus the .tgz single-directory variants) for users of older systems and older CPUs.

Ubuntu 24.04 only ships OpenSSL 3.0 and Python 3.12, and the 3.14 from actions/setup-python for it links against that OpenSSL 3.0. So the job:

  • builds OpenSSL 3.5.8 from source (no-tests no-docs, only make + make install_sw - the OpenSSL test suite is very slow and is never run), with an rpath to its prefix, so that its libssl and the openssl program do not pick up the system's OpenSSL 3.0 libraries with the same SONAMEs,
  • builds Python 3.14.7 from source against that OpenSSL (--enable-shared --enable-optimizations --with-openssl=... --with-openssl-rpath=auto, like the setup-python builds the other Linux binaries use; the PGO training run, a subset of python's test suite, runs in parallel worker processes),
  • builds borg with its crypto extension linked against that OpenSSL (BORG_OPENSSL_PREFIX + rpath), then the PyInstaller binary,
  • checks that it all fits together: the ssl module reports OpenSSL 3.5.8, borg's crypto extension links libcrypto from the prefix, the frozen borg-dir bundle contains byte-identical copies of the built libcrypto/libssl/libpython, borg.exe debug info reports CPython 3.14.7, and scripts/glibc_check.py asserts that no bundled ELF file needs more than glibc 2.39,
  • runs the test suite with borg.exe in PATH, so the binary-parametrized tests exercise the binary,
  • on tags: attests the provenance (after the tests passed) and hands the binaries and their sigstore bundles over to the release job, like the other binary builds. The artifact is uploaded on every run, so a master/PR binary can be tried out on an older system.

The sources are pinned by version and sha256 (bump both together). The checksums were verified against the .sha256 file published with the OpenSSL release and against the python.org sigstore bundle of the Python tarball.

Other changes:

  • the release job now also requires this job, release.yml expects the four new assets and the release notes mention the two Linux flavours,
  • scripts/glibc_check.py no longer crashes on an ELF file without versioned glibc symbols (it is used by the new job),
  • docs: binaries 00_README.txt, changelog, and the testsuite comment about when borg.exe gets built.

Notes for review:

  • The job runs on PRs too, like the other test jobs (roughly 40 min per arch, mostly the PGO Python build). Add if: github.event_name != 'pull_request' if that is too much.
  • arm64 is included because the existing Linux flavour ships both arches; drop the matrix entry if only x86_64 is wanted.
  • No SFTP/S3/rest-over-ssh test servers in this job: those tests exercise the transports, not the toolchain, and native_tests covers them.

🤖 Generated with Claude Code

@codecov

codecov Bot commented Sep 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.88%. Comparing base (b2b1a42) to head (7ae795e).
⚠️ Report is 1 commits behind head on master.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##           master   #10344   +/-   ##
=======================================
  Coverage   87.87%   87.88%           
=======================================
  Files         103      103           
  Lines       18878    18878           
  Branches     2916     2916           
=======================================
+ Hits        16589    16590    +1     
+ Misses       1589     1586    -3     
- Partials      700      702    +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

The Linux binaries from native_tests are built on ubuntu-26.04 and thus
need glibc 2.43, and the x86_64 one does not run on CPUs below x86-64-v3
("Illegal instruction"). The new oldglibc_binary job builds on
ubuntu-24.04 and ubuntu-24.04-arm (glibc 2.39, the oldest GitHub-hosted
Ubuntu image, with its baseline x86-64 toolchain and packages) and
produces borg-linux-glibc239-{x86_64,arm64}-gh.

Fixes borgbackup#10342.

Ubuntu 24.04 only has OpenSSL 3.0 and Python 3.12 (and setup-python's
3.14 for it links the system OpenSSL 3.0), so the job builds OpenSSL
3.5.8 (no-tests, no-docs - the OpenSSL test suite is very slow and is
never run) and Python 3.14.7 from source, pinned by version and sha256,
links Python and borg's crypto extension against that OpenSSL, and
PyInstaller bundles those libraries. OpenSSL is built with an rpath to
its prefix, so that its libssl and the openssl program do not pick up
the system's OpenSSL 3.0 libraries, which have the same SONAMEs. Python
is a PGO build like the setup-python ones; its PGO training run (a
subset of python's test suite) runs in parallel worker processes. The
job checks that the ssl module, borg's crypto extension and the frozen
binary really use the OpenSSL and Python built there and that no
bundled ELF file needs a newer glibc than the binary's name promises.

The binary is built on every run of the job and the test suite runs
against it. On tags, the binaries get a provenance attestation and the
release job (which now depends on this job) attaches them and their
sigstore bundles to the GitHub release.

Also: scripts/glibc_check.py no longer crashes on an ELF file without
versioned glibc symbols.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@ThomasWaldmann
ThomasWaldmann force-pushed the ci-linux-oldglibc-binary branch from 33a5414 to 7ae795e Compare September 9, 2026 16:03
@ThomasWaldmann ThomasWaldmann changed the title CI: also build Linux binaries for older glibc on ubuntu-24.04 CI: build Linux binaries for older CPUs/glibc on ubuntu-24.04 Sep 9, 2026
@ThomasWaldmann
ThomasWaldmann merged commit 8f6e355 into borgbackup:master Sep 9, 2026
26 checks passed
@ThomasWaldmann
ThomasWaldmann deleted the ci-linux-oldglibc-binary branch September 9, 2026 16:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

borg2 b24: precompiled binaries don't work on cpus < x86-64-v3

1 participant