[selzoc]

Compiled releases ship every package in the release, including toolchains like Go or Rust that are only needed to build other packages. No job uses them, and the director never installs them on a VM. They still take up space in every download, blobstore upload and export.

This adds an opt-in --exclude-compile-time-packages flag to bosh-agent compile. Everything is still compiled exactly as before; only the output tarball changes:

• Any package that a job lists is kept. The list is read from each job's own manifest inside the job tarball, because release.MF doesn't always include it.
• Other packages are left out of the tarball. They are still recorded in the release manifest under compile_time_packages, so you can see what was removed.
• If a kept package depended on a removed one, it gets a new fingerprint and version, with the old fingerprint saved alongside. The director then treats it as a different compiled package and never mixes it up with the untrimmed one. Its fingerprint is derived the same way the director keys dependencies, and tests check this against values produced by the director's own code.
• The release version gets a +trimmed suffix, or .trimmed if the version already has a +. If nothing is removed, the output is byte-for-byte the same as without the flag.
• Compilation fails with a clear error if a job's manifest is missing, duplicated or malformed, if a job lists a package the release doesn't have, or if no job lists any of the release's packages.

Without the flag, nothing changes.

This depends on the bosh-cli feature/compile-time-packages pr: cloudfoundry/bosh-cli#743, which adds the two new manifest fields and keeps them when releases are repacked or re-digested. go.mod points at that branch for now and should move to a bosh-cli release before this merges. Older CLIs and directors ignore the new fields.

How much it saves

We trimmed the exact compiled releases that cf-deployment and bosh-deployment reference today:

Deployment	Releases	Before	After	Saved
cf-deployment	31	5.46 GB	4.01 GB	1.45 GB (26.6%)
bosh-deployment	6	1.95 GB	1.33 GB	623 MB (31.9%)

About half of the releases have nothing to remove and come out unchanged.

create-env also gets faster. Unlike the director, it installs every package a job depends on, not just the ones the job lists. For a base director that drops from 19 packages to 15, about 400 MB less to download and 1.29 GB less to unpack.

Is it safe?

For normal deployments, yes: VMs only ever receive the packages their jobs list, and those are always kept. create-env installs more than that, so we also checked every affected release in both deployments for files that need something from a removed package. Nothing a job uses was affected. The one thing we found was in backup-and-restore-sdk: a test helper that needs a library no job declares. It is already broken in the untrimmed release, and the backup job itself works fine.

Testing

• Unit tests cover which packages are kept, the new identities, the version suffix, the manifest contents and each error.
• An optional test suite (go test -tags fixtures ./releasetarball/) runs real release tarballs through the compiler with a fake package build. It checks that every kept package's blob is present and matches its checksum.
• On a live director, using small test releases, we checked that:
  • source and trimmed releases upload in either order and coexist;
  • bumping a removed toolchain produces a new build of the app;
  • adding or removing a package from a job between versions deploys cleanly when upgrading in either order;
  • exporting a trimmed deployment and redeploying from the export works;
  • a trimmed release deploys on a different compatible stemcell.

To roll back, deploy the untrimmed version of the release.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Alphasite]

The test framework needs to be fixed, i think most of the rest of it is minor comments

[Alphasite]

ginkgo has a native way of writing prameterised tests which may be worth using here. This isnt awful, its just a lot of code in a describe body

[Alphasite]

Also we may hyst want to hard code the fixture names, i get it, but i dont imagine it changes so often as to be a problem