feat(runner): persistent mise/hk caches, nightly pruner, explicit ephemeral, image identity

[azlekov]

Stream S2 of the CI redesign: runner image changes. Targets this fork's main (not upstream).

Refs #16, #8. Partially addresses #7 (see below).

Changes

1. Persistent mise + hk caches. The Dockerfile sets MISE_DATA_DIR=/root/.local/share/mise, MISE_CACHE_DIR=/root/.cache/mise, HK_CACHE_DIR=/root/.cache/hk and HK_PKL_CACHE_DIR=/root/.cache/hk/pklr (hk's real env vars per the hk docs; pinning the pkl cache under HK_CACHE_DIR keeps it on the same volume). One named volume per service per directory. ~/.cache is never mounted as a whole, so it can't shadow the pnpm volume.
2. ~/.rustup and ~/.konan volumes per service (No ~/.rustup volume: the Rust toolchain reinstalls on every X64 job #8).
3. Pruner replaces per-job scans (cleanup.sh adds 20–39 s to every self-hosted job: du -sb + full find over ~20 GB caches on each job-completed hook #16). The lru_cap scans leave cleanup.sh. A new pruner compose service (same image, pruner.sh entrypoint, docker.sock plus only the capped cache volumes) runs daily: LRU caps on gradle, bun, mise cache and hk (ordered by atime, because Gradle never touches mtime on read), whole-tool/version eviction for mise installs, docker system prune -af --filter until=24h, docker builder prune -af --filter until=72h. Named volumes are never pruned. cleanup.sh keeps the cheap work: /tmp sweep, Gradle daemon/journal/locks, _diag. I also made two of its own walks cheap: the /tmp sweep no longer descends into the spared workdir (find -depth disables prune), and the Gradle lock find is limited to caches/ at depth 3.
4. Ephemeral made explicit. See below.
5. Image identity. ARG/ENV RUNNER_IMAGE_TAG and RUNNER_IMAGE_REVISION, set by deploy.yml (docker/metadata-action pinned tag, github.sha) and build.yml (pr-<n>, head SHA). The ARGs sit late in the Dockerfile so they don't bust the SDK layers.
6. Healthcheck per runner: pgrep -f Runner.Listener in exec form (as in closed feat: make the three-runner compose a Coolify service #10; under CMD-SHELL pgrep -f matches its own sh -c).
7. Per-host env examples hosts/{hetzner,beelink,rpi5}.env.example with the target labels, hosts/rpi5.compose.yml, LABELS default, README cache/label tables and rollout.

The ephemeral finding

I read /entrypoint.sh from myoung34/github-runner:2.338.0-ubuntu-noble: both flags are if [ -n "${EPHEMERAL}" ] / if [ -n "${DISABLE_AUTO_UPDATE}" ], so any non-empty value enables them. EPHEMERAL: "false" always passed --ephemeral: these runners have always been one-job-per-container, restarted by restart: unless-stopped. This PR sets EPHEMERAL: "true" with a comment, which changes no behaviour. Caches persist because they live on named volumes, not in the container. DISABLE_AUTO_UPDATE now defaults to empty (updates on); .env.example documents that any non-empty value, including "false", disables updates. (#10 took the opposite view, that omitting EPHEMERAL makes runners persistent; that is true of an unset variable but not of the "false" that was shipped.)

Expected per-job saving

Issue #16 measured the Complete runner hook at 11-41 s per job (median ~25 s), with the post-job disk-free unchanged. After this PR the hook does only a shallow /tmp sweep and a depth-limited Gradle find; expect roughly 1-3 s. That is about 20-35 s per self-hosted job, ~4-5 runner-hours a week fleet-wide. Persistent mise/hk caches additionally avoid re-downloading tools on every fresh container; I did not measure that.

Host rollout

Runners are ephemeral and re-register on their next job. On each host:

1. Update .env from hosts/<host>.env.example (keep real APP_* values), then docker compose pull && docker compose up -d.
2. hetzner: LABELS=self-hosted,Linux,X64,docker,x86,light,agent,hetzner (adds agent).
3. beelink: LABELS=self-hosted,Linux,X64,docker,x86,large,light,beelink (adds light).
4. rpi5: LABELS=self-hosted,Linux,ARM64,docker,agent,rpi5 (drops light, adds agent) and COMPOSE_FILE=docker-compose.yml:hosts/rpi5.compose.yml so no heavy cache volumes are mounted.
5. Check docker compose ps shows healthy, and the runner list shows the new labels.

New cache volumes start empty and warm on first jobs. Existing runner-work* volumes are untouched. jdx/mise-action should use cache: false on these runners.

Not done / notes

• cleanup.sh LRU-caps only gradle and bun; cargo, npm and pnpm are unbounded #7: only gradle, bun, mise and hk are capped. npm, pnpm, cargo, rustup and konan are still uncapped; pnpm/cargo hard-link into projects, so partial file eviction needs care.
• Container actions and docker run -v get empty mounts: runner work dir must be mounted at the same host path #19 (sparse-checkout heal in pre-job.sh) is out of scope.
• The pruner can evict from a cache while a job on that runner reads it; this only happens above the cap, and it evicts least recently used first.

Verification

• docker compose config valid for the base file and with -f hosts/rpi5.compose.yml (rpi5 runners then have only docker.sock and work; pruner has only docker.sock).
• bash -n and shellcheck clean on cleanup.sh, pruner.sh, pre-job.sh.
• Local image build (host arch) succeeds; RUNNER_IMAGE_TAG, RUNNER_IMAGE_REVISION, MISE_*, HK_* present in the image.
• pruner.sh and cleanup.sh run inside the image against fixture dirs: LRU cap evicted the oldest file and the oldest mise tool version; /tmp sweep kept only the workdir tree.
• Workflows parse as YAML. The workflows themselves (deploy/build) were not run.

🤖 Generated with Claude Code