Skip to content

Add SetupCcache: use the shared ccache stores from CI jobs - #49

Merged
graphite-app[bot] merged 1 commit into
mainfrom
setup-ccache
Oct 16, 2026
Merged

graphite-app[bot] merged 1 commit into
mainfrom
setup-ccache

Conversation

@willson556

@willson556 willson556 commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

Step 1 of the shared ccache store rollout (CI RFC Proposal 2). Both stores are up and verified: Stafl-EMB03 on the LAN for self-hosted runners, and ccache1.ci.emb.staflsystems.com on AWS for GitHub-hosted ones. This adds the action that points CI jobs at them. Merging it changes nothing until a workflow calls it with mode: remote.

What it does:

  • Picks the store from runner.environment: self-hosted runners use the LAN store, and GitHub-hosted runners use the AWS one. URLs and passwords come from the AWS_CCACHE_URL, LAN_CCACHE_URL, AWS_CCACHE_HTTP_PASSWORD and LAN_CCACHE_HTTP_PASSWORD org variables and secrets.
  • Tunnels to AWS over TLS: ccache's HTTP backend can't speak HTTPS. So on GitHub-hosted runners the action starts a ghostunnel client on localhost:8080 that forwards over TLS, checked against the system trust store. ghostunnel is pinned to v1.11.3 and its sha256 is verified.
  • Modes: mode: actions (the default) keeps today's Actions-cache behavior through hendrikmuhs/ccache-action. Callers pass vars.CCACHE_MODE, so setting it back to actions is the rollback. In remote mode, hendrikmuhs/ccache-action still runs, with restore and save off. That keeps its ccache -z at the start and ccache -s at the end, which CIMetrics --ccache (CIMetrics: report ccache hit rates with --ccache #48) reads.
  • write-only: 'true' sets ccache's recache, for pushes to main. In actions mode it skips the restore, as the BMS2000 repos' restore: github.ref != 'refs/heads/main' does today.
  • fork: 'true' installs the IAR-capable fork. It replaces the "setup ccache" step that is copied into every BMS2000 repo.
  • Fails open: if the store can't be reached, the job warns and builds without it.

ccache versions: the jobs run Ubuntu's 4.5.1 (CI image), 4.9 and the 4.12 fork. I checked the ccache source for how each one reads the store settings:

  • 4.5.1 names the setting secondary_storage, and 4.7 renamed it remote_storage but still accepts the old name. So the action sets CCACHE_SECONDARY_STORAGE.
  • remote_only only exists from 4.7. 4.5.1 errors on an unknown key in a config file but ignores an unknown CCACHE_* environment variable. So the action sets CCACHE_REMOTE_ONLY=1 in the environment, and nothing goes in ccache_config.conf.

Smoke test: CustomGithubActions is public, so it can't run one. The org's ccache variables and secrets are for private repositories only, and the self-hosted runner groups don't allow public repositories. The first run here got empty store settings on GitHub-hosted runners, and its ubuntu-iar job was never picked up. So:

For each store and ccache version, it builds 20 run-unique files three times, each in its own step, which also checks that the tunnel outlives the step that started it:

  1. All misses, with no store errors.
  2. After ccache -C, every file hits, and the store served at least one hit per file. ccache 4.5.1 counts a manifest and an object hit per file, so it reports 40.
  3. With the store unreachable, the build must still pass.

Smoke results (run 38816301601): all three jobs passed all three builds.

  • AWS, ccache 4.9.1: 20/20 hits from the store, through a tunnel that outlived its step.
  • AWS, ccache 4.5.1: 20/20 hits. CCACHE_SECONDARY_STORAGE works and CCACHE_REMOTE_ONLY is ignored, as expected.
  • LAN (10.10.1.22), IAR fork (4.12): 20/20 hits from Stafl-EMB03.
  • With the store unreachable, every job built with all misses and passed.

An earlier LAN run failed because Stafl-EMB03 hadn't picked up its DHCP reservation: its DHCP client ID wasn't its MAC. With dhcp-identifier: mac in its netplan config, it's now at 10.10.1.22, which is what LAN_CCACHE_URL points to.

Next: once this passes, the A/B (rollout step 2): StaflLib's host-Debug and arm-cm7-iar-Debug pass mode: remote.

🤖 Generated with Claude Code

@willson556
willson556 changed the base branch from main to ci-status October 15, 2026 20:48

willson556 commented Oct 15, 2026 •

Copy link
Copy Markdown
Member Author

How to use the Graphite Merge Queue

Add the label merge-queue to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

@willson556
willson556 marked this pull request as ready for review October 15, 2026 21:28
@staflsystemsci
staflsystemsci Bot requested review from a team, fcheein, gretl-silva-stafl, mament0, mns4595, mzhu48, nehalkpatel and thebhef and removed request for a team October 15, 2026 21:28
@staflsystemsci

staflsystemsci Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Review SLA

Nobody owes a review on this PR right now.

Due times are in business hours, 10:00 to 17:00 Pacific: 4 h for a PR under 250 added lines, 7 h otherwise, and 4 h for a re-review. Mid-stack, an assignee owes a response on the lowest PR they haven't approved. Dashboard

@graphite-app

graphite-app Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Merge activity

Step 1 of the shared ccache store rollout (CI RFC Proposal 2). Both stores are up and verified: Stafl-EMB03 on the LAN for self-hosted runners, and `ccache1.ci.emb.staflsystems.com` on AWS for GitHub-hosted ones. This adds the action that points CI jobs at them. Merging it changes nothing until a workflow calls it with `mode: remote`.

**What it does:**
- **Picks the store from `runner.environment`:** self-hosted runners use the LAN store, and GitHub-hosted runners use the AWS one. URLs and passwords come from the `AWS_CCACHE_URL`, `LAN_CCACHE_URL`, `AWS_CCACHE_HTTP_PASSWORD` and `LAN_CCACHE_HTTP_PASSWORD` org variables and secrets.
- **Tunnels to AWS over TLS:** ccache's HTTP backend can't speak HTTPS. So on GitHub-hosted runners the action starts a ghostunnel client on `localhost:8080` that forwards over TLS, checked against the system trust store. ghostunnel is pinned to v1.11.3 and its sha256 is verified.
- **Modes:** `mode: actions` (the default) keeps today's Actions-cache behavior through `hendrikmuhs/ccache-action`. Callers pass `vars.CCACHE_MODE`, so setting it back to `actions` is the rollback. In `remote` mode, `hendrikmuhs/ccache-action` still runs, with restore and save off. That keeps its `ccache -z` at the start and `ccache -s` at the end, which `CIMetrics --ccache` (#48) reads.
- **`write-only: 'true'`** sets ccache's `recache`, for pushes to main. In `actions` mode it skips the restore, as the BMS2000 repos' `restore: github.ref != 'refs/heads/main'` does today.
- **`fork: 'true'`** installs the IAR-capable fork. It replaces the "setup ccache" step that is copied into every BMS2000 repo.
- **Fails open:** if the store can't be reached, the job warns and builds without it.

**ccache versions:** the jobs run Ubuntu's 4.5.1 (CI image), 4.9 and the 4.12 fork. I checked the ccache source for how each one reads the store settings:
- 4.5.1 names the setting `secondary_storage`, and 4.7 renamed it `remote_storage` but still accepts the old name. So the action sets `CCACHE_SECONDARY_STORAGE`.
- `remote_only` only exists from 4.7. 4.5.1 errors on an unknown key in a config file but ignores an unknown `CCACHE_*` environment variable. So the action sets `CCACHE_REMOTE_ONLY=1` in the environment, and nothing goes in `ccache_config.conf`.

**Smoke test:** CustomGithubActions is public, so it can't run one. The org's ccache variables and secrets are for private repositories only, and the self-hosted runner groups don't allow public repositories. The first run here got empty store settings on GitHub-hosted runners, and its `ubuntu-iar` job was never picked up. So:
- `ci.yml` here runs `shellcheck` on `SetupCcache/` (stacked on #52, which adds `ci.yml` and `CI Status`).
- The end-to-end test lives in the private [StaflSystems/ci-smoke-tests](https://github.com/StaflSystems/ci-smoke-tests), which checks out this repo at a given ref: `gh workflow run setup-ccache-smoke.yml -R StaflSystems/ci-smoke-tests -f ref=setup-ccache`.

For each store and ccache version, it builds 20 run-unique files three times, each in its own step, which also checks that the tunnel outlives the step that started it:
1. All misses, with no store errors.
2. After `ccache -C`, every file hits, and the store served at least one hit per file. ccache 4.5.1 counts a manifest and an object hit per file, so it reports 40.
3. With the store unreachable, the build must still pass.

**Smoke results** ([run 38816301601](https://github.com/StaflSystems/ci-smoke-tests/actions/runs/38816301601)): all three jobs passed all three builds.
- AWS, ccache 4.9.1: 20/20 hits from the store, through a tunnel that outlived its step.
- AWS, ccache 4.5.1: 20/20 hits. `CCACHE_SECONDARY_STORAGE` works and `CCACHE_REMOTE_ONLY` is ignored, as expected.
- LAN (10.10.1.22), IAR fork (4.12): 20/20 hits from Stafl-EMB03.
- With the store unreachable, every job built with all misses and passed.

An earlier LAN run failed because Stafl-EMB03 hadn't picked up its DHCP reservation: its DHCP client ID wasn't its MAC. With `dhcp-identifier: mac` in its netplan config, it's now at 10.10.1.22, which is what `LAN_CCACHE_URL` points to.

**Next:** once this passes, the A/B (rollout step 2): StaflLib's `host-Debug` and `arm-cm7-iar-Debug` pass `mode: remote`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Base automatically changed from ci-status to main October 16, 2026 17:33
@graphite-app
graphite-app Bot merged commit 19bccac into main Oct 16, 2026
7 checks passed
@graphite-app
graphite-app Bot deleted the setup-ccache branch October 16, 2026 17:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants