binaries: strip debug symbols on Linux, build cryptography against the bundled OpenSSL - #10346
Merged
ThomasWaldmann merged 1 commit intoSep 9, 2026
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #10346 +/- ##
==========================================
- Coverage 87.88% 85.34% -2.54%
==========================================
Files 103 103
Lines 18878 18878
Branches 2916 2916
==========================================
- Hits 16590 16112 -478
- Misses 1586 2078 +492
+ Partials 702 688 -14 ☔ View full report in Codecov by Harness. |
…odels, one OpenSSL The Linux binaries were about 70 MiB as a single file and unpacked to 152 MiB at every start, see borgbackup#10345. Three changes: Strip the debug symbols on Linux: pythons built with the default CFLAGS carry -g, and the spec had strip=False everywhere, so about half of the Linux binaries was debug info. Enable PyInstaller's strip for the collected shared libraries on Linux (only there: on macOS it would interfere with code signing, and it is not recommended on Windows). Python tracebacks do not need the symbols, only gdb would. Measured on the glibc239 x86_64 bundle: 152 -> 78 MiB unpacked, 69.6 -> 43.4 MiB compressed. Bundle only the botocore service models borg needs: botocore (boto3, s3 extra) ships the JSON service models of all ~430 AWS services, 23 MiB, about 14 MiB in the compressed binary. borg only ever creates an s3 client - also for S3-compatible services like MinIO or Backblaze B2, which just get a different endpoint URL - so the spec keeps only the s3 model, the shared top-level files and the models the credential providers may need (sts, sso, sso-oidc). In the oldglibc_binary job, build cryptography (a paramiko dependency from the sftp extra) from source against the OpenSSL built there, instead of using the manylinux wheel that statically links its own copy of OpenSSL - the binary contained two OpenSSLs. The runner image has the Rust toolchain this needs. The job now checks that the bundled OpenSSL and Python libraries are the ones built there by their GNU build IDs (the files are stripped now), that cryptography links and loads that OpenSSL, that no bundled shared library still carries debug info, that the other botocore models are gone, prints the bundle sizes, and smoke-tests the frozen binary against a MinIO server - the test suite's S3 test runs borg from the venv, not the binary. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
ThomasWaldmann
force-pushed
the
binary-size-10345
branch
from
September 9, 2026 19:11
6f9cb88 to
9f3f692
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #10346 +/- ##
=======================================
Coverage 87.88% 87.88%
=======================================
Files 103 103
Lines 18878 18878
Branches 2916 2916
=======================================
Hits 16590 16590
Misses 1586 1586
Partials 702 702 ☔ View full report in Codecov by Harness. |
ThomasWaldmann
added a commit
that referenced
this pull request
Sep 9, 2026
…43 Linux binaries #10346 did this only in the oldglibc_binary job (glibc239 binaries). The Linux binaries from native_tests (ubuntu-26.04, glibc243) still bundled two OpenSSLs: the system's libcrypto, which borg's crypto extension uses and PyInstaller bundles, and the copy statically linked into the manylinux wheel of cryptography (a paramiko dependency, sftp extra). Build cryptography from source against the system OpenSSL there, too, for the Linux "binary" matrix entries - the runner image has the Rust toolchain. The step checks that the extension links libcrypto dynamically and loads the same OpenSSL version as the system's openssl program; the smoke test of the built binary checks that the bundled extension links the bundled libcrypto, and prints the bundle sizes. The stripping of the debug symbols and the trimming of the botocore models from #10346 live in the shared PyInstaller spec, so they already apply to these binaries. See #10345. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The first three items of #10345.
Strip the debug symbols on Linux (
scripts/borg.exe.spec): pythons built with the defaultCFLAGScarry-g, and the spec hadstrip=Falseeverywhere, so about half of the Linux binaries was debug info. PyInstaller's strip now runs on the collected shared libraries (libpython, the extension modules, libcrypto, ...) on Linux only: on macOS it would interfere with code signing, and it is not recommended on Windows. Python tracebacks do not need the symbols, only gdb would.Measured on the glibc239 x86_64 bundle of the last CI run of #10344 by stripping the shared libraries the same way: 152 -> 78 MiB unpacked, 69.6 -> 43.4 MiB compressed. A stripped bundle was tried on Debian 13 (repo-create with
aes256-ocb, create, list, check, extract). This applies to the glibc243 binaries as well, they share the spec.Build cryptography against the bundled OpenSSL (
oldglibc_binaryjob): cryptography (a paramiko dependency from the sftp extra) comes as a manylinux wheel that statically links its own copy of OpenSSL, so the binary contained two OpenSSLs. The job now builds it from source (OPENSSL_DIRpointing at the OpenSSL built in the job, an rpath viaRUSTFLAGSso it loads that one at runtime, same SONAME hazard as for python and borg) - the runner image has the Rust toolchain. Expected saving: a few MiB compressed; the main point is one OpenSSL in the binary, i.e. one thing to update on an OpenSSL CVE.New checks in the job: the extension links and loads the job's OpenSSL, the bundled one links libcrypto dynamically, no bundled shared library still carries debug info, and the bundle sizes are printed in the log.
Bundle only the botocore service models borg needs (
scripts/borg.exe.spec): botocore ships the JSON service models of all ~430 AWS services, 23 MiB, about 14 MiB in the compressed binary. borg only ever creates ans3client, also for S3-compatible services like MinIO or Backblaze B2 (they just get a different endpoint URL), so the spec now keeps thes3model, the shared top-level files (endpoints.json,partitions.json,_retry.json,sdk-default-configuration.json) and the models the credential providers may need:sts(assume-role and web-identity profiles),ssoandsso-oidc(SSO profiles). Theoldglibc_binaryjob checks that the other models are gone and smoke-tests the frozen binary against a MinIO server (repo-create, create, list, check, extract, repo-delete): the test suite's S3 test runs borg from the venv, not the binary.🤖 Generated with Claude Code