From 13dad7a61af2780750be1411bcc0bfd330fc4c6a Mon Sep 17 00:00:00 2001 From: Artjoms Stukans Date: Sun, 15 Nov 2026 01:09:44 +0100 Subject: [PATCH 1/4] Design release.yml publishing for the factory images A v* tag should publish the run worker and the five run-unit images like the four services: versioned, multi-arch, signed, with an SBOM. Records the catalogue handoff for the agent images and the gate fixes. Part of #391 --- ...026-11-15-release-factory-images-design.md | 40 +++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 docs/superpowers/specs/2026-11-15-release-factory-images-design.md diff --git a/docs/superpowers/specs/2026-11-15-release-factory-images-design.md b/docs/superpowers/specs/2026-11-15-release-factory-images-design.md new file mode 100644 index 00000000..5386729a --- /dev/null +++ b/docs/superpowers/specs/2026-11-15-release-factory-images-design.md @@ -0,0 +1,40 @@ +# Release the factory images on a version tag + +Follow-up to [the GHCR factory images design](2026-11-14-ghcr-factory-images-design.md), which put the run worker +and the five run-unit images on GHCR as `:edge` and `:sha-` from `docker.yml` and left `release.yml` out of +scope. Goal: a `v*` tag publishes them the way it publishes the four services, so an operator can pin +`SPIRE_VERSION=` for every image `compose.ghcr.yml` names. + +## Decisions + +1. **One `publish` matrix, ten images.** `release.yml`'s matrix gains `spire-run-worker`, `spire-publisher`, + `spire-model-proxy`, `spire-egress-proxy`, `spire-agent-codex` and `spire-agent-claude-code`. Each is built by + `build-push-action` for `linux/amd64,linux/arm64`, pushed as `:` and `:latest`, with provenance and an + SBOM, and signed keyless by digest, exactly as the four services are. The same `gate` job holds all of them. +2. **The matrix mirrors `docker.yml`'s fields** (`apkUpgrade`, `gradleDist`), so `APK_UPGRADE_BUST=github.run_id` + reaches every apk-based image and the publisher and model proxy run `installDist` first. +3. **The agent images get their model catalogue from their own build scripts.** A multi-arch push cannot run the + image it builds, so the job first runs `deploy/agent/build-codex.sh` (or `build-claude-code.sh`) on the runner's + default builder, amd64, reads `dev.codespire.agent.models` back from that image, and passes it to the multi-arch + build as `AGENT_MODELS`. One recipe again: the trim logic stays in the script. The step runs before + `setup-buildx`, because a docker-container builder would not load the image the script then reads. + The catalogue comes from the amd64 binary; the arm64 binary is the same CLI version from the same npm release. +4. **Every image is multi-arch.** Checked statically: the JVM images use `eclipse-temurin:25-jre-alpine`, which the + four services already publish for arm64; the egress proxy uses `alpine:3.20`; the agents use + `node:22-bookworm-slim` (amd64 and arm64), and both CLIs ship a `linux-arm64` optional dependency on npm + (`@openai/codex@0.156.1`: `@openai/codex-linux-arm64`; `@anthropic-ai/claude-code@2.1.319`: + `@anthropic-ai/claude-code-linux-arm64`). +5. **The spire-arch checks read both workflows.** `ApkUpgradeIsNotCachedTest` holds `release.yml`'s matrix to the + same two directions as `docker.yml`'s, and a new check fails when the two workflows publish different image sets. + +## Gate fixes + +- The gate's `.env` lacked `SPIRE_OIDC_RUN_WORKER_SECRET`, which `compose.ghcr.yml` requires with `:?` on the + profiled run worker; compose interpolates every service before it filters by profile, so `up` refused to start + (`e2e.yml` already carries the value for this reason). The last dry run (2026-08-05) predates the requirement. +- The gate still builds and starts only the reviewer stack at `:edge`. It does not start the factory profile; it + now also renders `--profile factory` config, so a variable the run worker requires fails the gate. + +## Out of scope + +The Kubernetes deployment; live proof of the run worker image and the Claude Code image (docs/UNVERIFIED.md). From f356c5fb7f4c3460d8d2b449b70bd19377ff7ee5 Mon Sep 17 00:00:00 2001 From: Artjoms Stukans Date: Sun, 15 Nov 2026 01:11:43 +0100 Subject: [PATCH 2/4] Publish the run worker and factory images on a release tag release.yml now publishes all ten images docker.yml publishes as :edge, multi-arch, signed and with an SBOM. The agent images read their model catalogue from their own build scripts on amd64 first. Every apk-based image gets APK_UPGRADE_BUST. The gate's .env gains the run worker's required OIDC secret, and the gate renders the factory profile so a missing variable fails it. ApkUpgradeIsNotCachedTest reads both workflows, and a new check fails when release.yml and docker.yml publish different images. Part of #391 --- .github/workflows/release.yml | 128 +++++++++++++++++- .../arch/ApkUpgradeIsNotCachedTest.java | 56 +++++--- .../codespire/arch/DockerBuildSources.java | 26 +++- .../DockerWorkflowBuildsEveryImageTest.java | 25 +++- 4 files changed, 207 insertions(+), 28 deletions(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 274f5d09..dbb97bf3 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -7,6 +7,10 @@ # arm64 goes through QEMU and costs tens of minutes on a JVM image, which is exactly why this is # tag-triggered rather than on any per-merge path. # +# It publishes every image docker.yml publishes as :edge, the run worker and the run unit's images +# included, so SPIRE_VERSION= pins every image compose.ghcr.yml names. Until 2026-11-15 it +# published the four reviewer images only, and a version tag left the factory on whatever it last pulled. +# # It can also be run by hand, and a manual run is ALWAYS a dry run. That exists because this workflow # is otherwise untestable: it triggers only on a tag, so nothing that changes it — a bumped action, an # edited step — is exercised until a real release is already under way, which is the worst possible @@ -78,6 +82,10 @@ jobs: echo "SPIRE_OIDC_ORCHESTRATOR_SECRET=dev-only-orchestrator-secret" echo "SPIRE_OIDC_GATEWAY_SECRET=dev-only-gateway-secret" echo "SPIRE_OIDC_WORKER_SECRET=dev-only-worker-secret" + # compose interpolates every service before it filters by profile, so the run worker's + # required secret must exist although the gate never starts the factory profile. Without + # it `up` refuses the whole file; e2e.yml carries the same line for the same reason. + echo "SPIRE_OIDC_RUN_WORKER_SECRET=dev-only-run-worker-secret" echo "KEYCLOAK_ADMIN_USER=dev-admin" echo "KEYCLOAK_ADMIN_PASSWORD=TEST-keycloak-admin" echo "SPIRE_TRUSTED_PROXIES=172.16.0.0/12" @@ -85,6 +93,13 @@ jobs: } > deploy/.env grep -qE '^SPIRE_ENCRYPTION_KEYSET=.+' deploy/.env || { echo "keyset generation produced nothing"; exit 1; } + # The gate starts the reviewer stack only: the factory profile mounts the host's Docker socket and + # its run units need a forge and a model to do anything. This step still renders the file WITH that + # profile, so a variable the run worker requires, or a unit image reference that no longer + # interpolates at SPIRE_VERSION, fails the gate rather than the first operator who enables it. + - name: The factory profile interpolates + run: docker compose -f deploy/compose.ghcr.yml --env-file deploy/.env --profile factory config -q + - name: Up run: docker compose -f deploy/compose.ghcr.yml --env-file deploy/.env up -d @@ -119,25 +134,129 @@ jobs: strategy: fail-fast: false matrix: + # Every image docker.yml publishes as :edge, published here at the version: the four services, + # the run worker, the dashboard, the run unit's three companions and the two reference agents. + # DockerWorkflowBuildsEveryImageTest fails when the two lists differ, because an image left out + # here is one an operator cannot pin to SPIRE_VERSION= while compose.ghcr.yml names it. + # + # The fields mean what they mean in docker.yml. apkUpgrade sends APK_UPGRADE_BUST (docker.yml + # says at length why it is a security control); gradleDist names the module whose installDist + # must run first. catalogueScript names the agent image's build script, which reads the model + # catalogue: see the Model catalogue step. + # + # Keep no blank line and no comment between entries: ApkUpgradeIsNotCachedTest parses this + # list as source text. include: - name: spire-gateway context: . dockerfile: Dockerfile service: gateway + apkUpgrade: true + gradleDist: '' + catalogueScript: '' - name: spire-orchestrator context: . dockerfile: Dockerfile service: orchestrator + apkUpgrade: true + gradleDist: '' + catalogueScript: '' - name: spire-review-worker context: . dockerfile: Dockerfile service: review-worker + apkUpgrade: true + gradleDist: '' + catalogueScript: '' + - name: spire-run-worker + context: . + dockerfile: Dockerfile + service: run-worker + apkUpgrade: true + gradleDist: '' + catalogueScript: '' - name: spire-ui context: ./spire-ui dockerfile: ./spire-ui/Dockerfile service: '' + apkUpgrade: true + gradleDist: '' + catalogueScript: '' + - name: spire-publisher + context: ./spire-publisher + dockerfile: ./spire-publisher/Dockerfile + service: '' + apkUpgrade: true + gradleDist: spire-publisher + catalogueScript: '' + - name: spire-model-proxy + context: ./spire-model-proxy + dockerfile: ./spire-model-proxy/Dockerfile + service: '' + apkUpgrade: true + gradleDist: spire-model-proxy + catalogueScript: '' + - name: spire-egress-proxy + context: ./deploy/egress + dockerfile: ./deploy/egress/Dockerfile + service: '' + apkUpgrade: true + gradleDist: '' + catalogueScript: '' + - name: spire-agent-codex + context: ./deploy/agent + dockerfile: ./deploy/agent/codex/Dockerfile + service: '' + apkUpgrade: false + gradleDist: '' + catalogueScript: deploy/agent/build-codex.sh + - name: spire-agent-claude-code + context: ./deploy/agent + dockerfile: ./deploy/agent/claude-code/Dockerfile + service: '' + apkUpgrade: false + gradleDist: '' + catalogueScript: deploy/agent/build-claude-code.sh steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + + # The agent images carry their model catalogue as a label, and a label cannot come from a RUN's + # output (build-codex.sh says why). In docker.yml the script builds, reads and rebuilds; a + # multi-arch push cannot run what it builds, so here the script builds the amd64 image on the + # runner, the label is read back from it, and the value goes to the multi-arch build below as + # AGENT_MODELS. Calling the script keeps one recipe: the trim of `codex debug models` is not + # restated here, and a vendor change to that command fails the release as it fails docker.yml. + # The arm64 image installs the same pinned CLI release, so one catalogue describes both. + # + # Before setup-buildx, on purpose: that action makes its docker-container builder the default, + # which would not load the script's image into the local store, and the read would find none. + # The Claude Code script reads a file rather than the binary; it goes the same way so both + # agents' catalogues come from the scripts an operator runs. + - name: Model catalogue + id: catalogue + if: matrix.catalogueScript != '' + run: | + bash "${{ matrix.catalogueScript }}" "release-catalogue/${{ matrix.name }}:amd64" + models="$(docker inspect --format '{{ index .Config.Labels "dev.codespire.agent.models" }}' \ + "release-catalogue/${{ matrix.name }}:amd64")" + if [ -z "$models" ] || [ "$models" = "" ]; then + echo "the build script produced an image with no model catalogue"; exit 1 + fi + echo "models=$models" >> "$GITHUB_OUTPUT" + + # Same JDK and Gradle setup as docker.yml, for the two images that copy a distribution. The + # distribution is JVM bytecode, so one installDist serves both architectures. + - if: matrix.gradleDist != '' + uses: actions/setup-java@77dd5b2ce225ea11469928ac317850245a74ae78 # v6.2.0 + with: + distribution: temurin + java-version: '25' + - if: matrix.gradleDist != '' + uses: gradle/actions/setup-gradle@8da46b539cd8fe6ab27ca75128919a18d757604c # v6.4.1 + - name: Distribution + if: matrix.gradleDist != '' + run: ./gradlew --no-daemon --console=plain ":${{ matrix.gradleDist }}:installDist" + - uses: docker/setup-qemu-action@99012661954931238ded8c8b007157a8430204e1 # v4.4.0 - uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1 - uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0 @@ -168,7 +287,14 @@ jobs: with: context: ${{ matrix.context }} file: ${{ matrix.dockerfile }} - build-args: ${{ matrix.service && format('SERVICE={0}', matrix.service) || '' }} + # One line per argument, each empty for the entries it does not apply to; the action skips + # empty items. APK_UPGRADE_BUST is docker.yml's security control: without a per-build value + # the upgrade layer is restored from the gha cache docker.yml fills, and a release would ship + # the OS packages of whichever build first wrote that layer. + build-args: | + ${{ matrix.service && format('SERVICE={0}', matrix.service) || '' }} + ${{ matrix.apkUpgrade && format('APK_UPGRADE_BUST={0}', github.run_id) || '' }} + ${{ steps.catalogue.outputs.models && format('AGENT_MODELS={0}', steps.catalogue.outputs.models) || '' }} # Both architectures even on a dry run: arm64 is what exercises QEMU, so dropping it would # make the dry run silent about the one action it is most needed to check. platforms: linux/amd64,linux/arm64 diff --git a/spire-arch/src/test/java/dev/codespire/arch/ApkUpgradeIsNotCachedTest.java b/spire-arch/src/test/java/dev/codespire/arch/ApkUpgradeIsNotCachedTest.java index 0f5eec77..d7b3426e 100644 --- a/spire-arch/src/test/java/dev/codespire/arch/ApkUpgradeIsNotCachedTest.java +++ b/spire-arch/src/test/java/dev/codespire/arch/ApkUpgradeIsNotCachedTest.java @@ -84,45 +84,63 @@ void everyImageThatUpgradesOsPackagesBustsTheLayerCache() throws IOException { } @Test - void theWorkflowSendsAValueThatDiffersPerBuild() throws IOException { - String workflow = RootBuild.read(".github/workflows/docker.yml"); - - assertTrue(workflow.contains(BUST + "={0}', github.run_id"), - "docker.yml no longer passes " + BUST + " from github.run_id. A constant value keys " - + "the upgrade layer the same way on every build, which is the original defect " - + "with an argument added. github.run_id is unique per run and already in " - + "scope."); + void everyPublishingWorkflowSendsAValueThatDiffersPerBuild() throws IOException { + List constant = new ArrayList<>(); + for (String workflowPath : DockerBuildSources.PUBLISHING_WORKFLOWS) { + if (!RootBuild.read(workflowPath).contains(BUST + "={0}', github.run_id")) { + constant.add(workflowPath); + } + } + + assertEquals(List.of(), constant, + "these workflows no longer pass " + BUST + " from github.run_id. A constant value, or " + + "none, keys the upgrade layer the same way on every build, which is the " + + "original defect. github.run_id is unique per run and already in scope."); } + /** + * Both workflows, because each spells its build args separately: docker.yml publishes :edge and + * release.yml the versioned images, from the same gha layer cache. release.yml sent no + * {@code APK_UPGRADE_BUST} until 2026-11-15, so a release restored the upgrade layer docker.yml + * had cached and shipped whatever packages that build saw, while this check read docker.yml only. + */ @Test void everyBuiltImageIsSentTheArgumentExactlyWhenItConsumesIt() throws IOException { Set upgrading = DockerBuildSources.dockerfilesThatUpgradeOsPackages().keySet(); - Map built = DockerBuildSources.builtImages(); + List wrong = new ArrayList<>(); + for (String workflowPath : DockerBuildSources.PUBLISHING_WORKFLOWS) { + wrong.addAll(misdeclaredImages(workflowPath, upgrading)); + } + assertEquals(List.of(), wrong, + "a workflow's matrix disagrees with the Dockerfiles it builds"); + } + + private static List misdeclaredImages(String workflowPath, Set upgrading) throws IOException { + Map built = DockerBuildSources.builtImages(workflowPath); assertFalse(built.isEmpty(), - "no `- name:` entry was parsed out of docker.yml's build matrix. An empty matrix " - + "satisfies every assertion below, so failing to parse is an error here " + "no `- name:` entry was parsed out of " + workflowPath + "'s build matrix. An empty " + + "matrix satisfies every assertion, so failing to parse is an error here " + "rather than an empty answer."); - String workflow = RootBuild.read(".github/workflows/docker.yml"); + String workflow = RootBuild.read(workflowPath); List wrong = new ArrayList<>(); for (Map.Entry image : built.entrySet()) { boolean consumes = upgrading.contains(image.getValue()); boolean sent = DockerBuildSources.entryBody(workflow, image.getKey()).contains("apkUpgrade: true"); if (consumes && !sent) { - wrong.add(image.getKey() + " builds " + wrong.add(workflowPath + ": " + image.getKey() + " builds " + RootBuild.repoRoot().relativize(image.getValue()) + ", which consumes " + BUST + ", but its matrix entry sets no " + "`apkUpgrade: true` — the build-args line then renders empty and the " + "upgrade layer caches forever"); } else if (!consumes && sent) { - wrong.add(image.getKey() + " is sent " + BUST + " but its Dockerfile never consumes " - + "it — Docker warns about an unconsumed build arg on every build, and a " - + "warning nobody can act on trains people past the ones they can"); + wrong.add(workflowPath + ": " + image.getKey() + " is sent " + BUST + " but its " + + "Dockerfile never consumes it — Docker warns about an unconsumed build arg on " + + "every build, and a warning nobody can act on trains people past the ones " + + "they can"); } } - - assertEquals(List.of(), wrong, - "docker.yml's matrix disagrees with the Dockerfiles it builds"); + return wrong; } } diff --git a/spire-arch/src/test/java/dev/codespire/arch/DockerBuildSources.java b/spire-arch/src/test/java/dev/codespire/arch/DockerBuildSources.java index 5db883eb..5146a06b 100644 --- a/spire-arch/src/test/java/dev/codespire/arch/DockerBuildSources.java +++ b/spire-arch/src/test/java/dev/codespire/arch/DockerBuildSources.java @@ -17,8 +17,9 @@ import static org.junit.jupiter.api.Assertions.assertTrue; /** - * Source-text readers shared by the checks on docker.yml's build matrix and the Dockerfiles it - * builds: the matrix parser and the scan for Dockerfiles that upgrade their base OS packages. + * Source-text readers shared by the checks on docker.yml's and release.yml's build matrices and the + * Dockerfiles they build: the matrix parser and the scan for Dockerfiles that upgrade their base OS + * packages. */ final class DockerBuildSources { @@ -93,15 +94,28 @@ static List matrixIncludes(String workflow) { while (matrix.find()) { includes.add(matrix.group("entries")); } - assertFalse(includes.isEmpty(), "no `include:` block found in docker.yml's build matrix — " + assertFalse(includes.isEmpty(), "no `include:` block found in the workflow's build matrix — " + "the parser and the workflow disagree about its shape"); return includes; } - /** Image name to the absolute path of the Dockerfile docker.yml builds it from. */ - static Map builtImages() throws IOException { + /** The workflow that builds every image per merge and publishes {@code :edge}. */ + static final String DOCKER_WORKFLOW = ".github/workflows/docker.yml"; + + /** The workflow that publishes every image at a version on a {@code v*} tag. */ + static final String RELEASE_WORKFLOW = ".github/workflows/release.yml"; + + /** + * Both workflows that build published images. Each image's build args are spelled once per + * workflow, so a check on one alone let the other drift: release.yml sent no + * {@code APK_UPGRADE_BUST} at all until 2026-11-15 while docker.yml's matrix passed every check. + */ + static final List PUBLISHING_WORKFLOWS = List.of(DOCKER_WORKFLOW, RELEASE_WORKFLOW); + + /** Image name to the absolute path of the Dockerfile the given workflow builds it from. */ + static Map builtImages(String workflowPath) throws IOException { Map images = new LinkedHashMap<>(); - for (String include : matrixIncludes(RootBuild.read(".github/workflows/docker.yml"))) { + for (String include : matrixIncludes(RootBuild.read(workflowPath))) { Matcher entry = MATRIX_ENTRY.matcher(include); while (entry.find()) { Matcher file = DECLARED_DOCKERFILE.matcher(entry.group("body")); diff --git a/spire-arch/src/test/java/dev/codespire/arch/DockerWorkflowBuildsEveryImageTest.java b/spire-arch/src/test/java/dev/codespire/arch/DockerWorkflowBuildsEveryImageTest.java index c8fca1cb..20084363 100644 --- a/spire-arch/src/test/java/dev/codespire/arch/DockerWorkflowBuildsEveryImageTest.java +++ b/spire-arch/src/test/java/dev/codespire/arch/DockerWorkflowBuildsEveryImageTest.java @@ -7,6 +7,7 @@ import java.util.ArrayList; import java.util.LinkedHashSet; import java.util.List; +import java.util.Map; import java.util.Set; import static org.junit.jupiter.api.Assertions.assertEquals; @@ -22,7 +23,8 @@ class DockerWorkflowBuildsEveryImageTest { */ @Test void everyImageThatUpgradesOsPackagesIsBuiltByTheWorkflow() throws IOException { - Set built = new LinkedHashSet<>(DockerBuildSources.builtImages().values()); + Set built = new LinkedHashSet<>( + DockerBuildSources.builtImages(DockerBuildSources.DOCKER_WORKFLOW).values()); List unbuilt = new ArrayList<>(); for (Path file : DockerBuildSources.dockerfilesThatUpgradeOsPackages().keySet()) { String relative = RootBuild.repoRoot().relativize(file).toString().replace('\\', '/'); @@ -37,4 +39,23 @@ void everyImageThatUpgradesOsPackagesIsBuiltByTheWorkflow() throws IOException { + "layer caches forever wherever they are built. Add a matrix entry with " + "`apkUpgrade: true`."); } - } + + /** + * release.yml publishes every image docker.yml publishes, from the same Dockerfile. + * compose.ghcr.yml names each image at {@code SPIRE_VERSION}, so an image docker.yml publishes + * as {@code :edge} and release.yml does not is one an operator cannot pin to a version: the run + * worker and the five run-unit images were exactly that until 2026-11-15, and the docs had to + * say a version tag works only for some of them. + */ + @Test + void theReleasePublishesEveryImageDockerYmlPublishes() throws IOException { + Map edge = DockerBuildSources.builtImages(DockerBuildSources.DOCKER_WORKFLOW); + Map released = DockerBuildSources.builtImages(DockerBuildSources.RELEASE_WORKFLOW); + + assertEquals(edge, released, + "docker.yml and release.yml publish different images, or build one from different " + + "Dockerfiles. Every image in docker.yml's matrices needs an entry in " + + "release.yml's publish matrix, or a release ships a version tag that " + + "compose.ghcr.yml names and nothing pushed."); + } +} From ad424d9832058aa899663323e27db8f2103bb34e Mon Sep 17 00:00:00 2001 From: Artjoms Stukans Date: Sun, 15 Nov 2026 01:13:12 +0100 Subject: [PATCH 3/4] Document versioned releases of the factory images SPIRE_VERSION may now be a release version for every image, so the "sha- only" caveats go. Records the release half in CICD-AND-PACKAGING section 11, the 2026-11-14/15 live pull of four factory images in HISTORY, and what stays unproven in UNVERIFIED. Part of #391 --- CLAUDE.md | 18 ++++++++++------- deploy/.env.example | 6 +++--- deploy/README.md | 21 ++++++++++++++------ deploy/compose.ghcr.yml | 5 +++-- docs/CICD-AND-PACKAGING.md | 40 +++++++++++++++++++++++++++++++++++--- docs/HISTORY.md | 19 ++++++++++++++++++ docs/UNVERIFIED.md | 12 ++++++++++-- 7 files changed, 98 insertions(+), 23 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 7d19846f..7d75a5c5 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -274,11 +274,13 @@ its ticket told. M5 is next; harness reviews are being built alongside it. suite has not been re-measured since, and builds now run in CI rather than on the host. - **The automated GitLab run-unit gap is still open.** RunUnitSpec has no network field, so a run unit cannot reach the e2e stack's GitLab. A live GitHub run does not close it. -- **The factory images are published on merge to master but not yet live-pulled.** docker.yml builds the run - worker, publisher, model proxy, egress proxy, Codex agent and Claude Code agent as public GHCR packages (each links to - this public repository by its source label; the Claude Code one by operator decision 2026-11-15, removed if Anthropic - asks; ADR-032 amended 2026-11-14 and 2026-11-15); no deployment has run a unit from them yet, and they are amd64 - only, unsigned and without an SBOM. +- **The factory images are partly live-pulled.** docker.yml builds the run worker, publisher, model proxy, egress + proxy, Codex agent and Claude Code agent as public GHCR packages (each links to this public repository by its source + label; the Claude Code one by operator decision 2026-11-15, removed if Anthropic asks; ADR-032 amended 2026-11-14 and + 2026-11-15), and release.yml publishes them at a version, multi-arch and signed. On 2026-11-14/15 a dev-stack review + of spire-test #106 ran on the Codex agent, publisher, model proxy and egress proxy pulled at `sha-66dd409`; the + run worker and Claude Code images, the packaged run worker's Docker socket on Linux (uid 1001), any arm64 image and + a real release (no tag has run release.yml with them) are not live-proven (docs/UNVERIFIED.md). - **Claude Code ran on one model and one CLI version** (`claude-sonnet-5-5`, 2.1.319): its charge is not reconciled with the vendor, only the measured canaries are proven absent, and a re-review, a reply and a rate limit on it have no live run (docs/UNVERIFIED.md, the 3b entry). @@ -303,9 +305,11 @@ cd spire-ui && npm install && npm run dev # React dashboard :34000 (UI_PORT) ``` **The factory's images** are on GHCR (`ghcr.io/artyomsv/spire-{run-worker,publisher,model-proxy,egress-proxy,agent-codex,agent-claude-code}`, -`:edge` and `:sha-` from docker.yml; `compose.ghcr.yml` defaults to them). The Claude Code agent is public by operator +`:edge` and `:sha-` from docker.yml, amd64; `:` and `:latest` from release.yml on a `v*` tag, amd64 + arm64, +signed; `compose.ghcr.yml` defaults to them). The Claude Code agent is public by operator decision and removed if Anthropic asks (its CLI's terms do not address redistribution; LICENSING.md). The run worker pulls a unit image only when its daemon -lacks it, so pin `SPIRE_VERSION` or pull `edge` by hand on upgrade. For the dev stack and `compose.yml` they are still +lacks it, so pin `SPIRE_VERSION` (a release version or `sha-`) or pull `edge` by hand on upgrade. release.yml's +manual run is a dry run (builds both architectures, pushes and signs nothing). For the dev stack and `compose.yml` they are still built locally (SMOKE-TEST Mode Q, Mode R §7): ```bash diff --git a/deploy/.env.example b/deploy/.env.example index 40f06be4..4000e975 100644 --- a/deploy/.env.example +++ b/deploy/.env.example @@ -162,9 +162,9 @@ SPIRE_TRUSTED_PROXIES=172.16.0.0/12 SPIRE_PUBLIC_HOST= # --- Image tag --- -# `edge` tracks master. Pin a commit (sha-, every master build publishes one) for a -# deployment you intend to keep. A version tag works for the run worker and the unit images only once -# release.yml publishes them (a follow-up PR); until then pin sha-. compose.ghcr.yml takes the run unit's images +# `edge` tracks master. For a deployment you intend to keep, pin a release version (the tag without +# its v, e.g. 1.2.3: release.yml publishes every image at it, multi-arch and signed) or a commit +# (sha-, every master build publishes one, amd64 only). compose.ghcr.yml takes the run unit's images # at this tag too, and the run worker pulls a unit image only when its daemon lacks it: on `edge` # an upgrade must pull them by hand (deploy/README.md), or units keep running the first `edge` the # daemon ever pulled beside a newer run worker. diff --git a/deploy/README.md b/deploy/README.md index 3b7180b4..ecb3186f 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -208,10 +208,19 @@ service being refused by another, and the gateway's live database privileges. ## Images -`:edge` tracks `master`, and every master build also publishes `:sha-`; pin a `sha-` -tag for anything you intend to keep (a version tag works for the run worker and the unit images only once -`release.yml` publishes them, in a follow-up PR). Each image carries its licence as an OCI label and as a -file (LICENSING.md, "Published images"). +`:edge` tracks `master`, and every master build also publishes `:sha-` (both amd64 only). A `v*` +tag publishes every image below as `:` (the tag without its `v`) and `:latest`, for amd64 and arm64, +cosign-signed by digest with an SBOM and provenance (`release.yml`). For anything you intend to keep, pin +`SPIRE_VERSION` to a release version, or to a `sha-` between releases. Each image carries its +licence as an OCI label and as a file (LICENSING.md, "Published images"). + +To check a release image's signature (keyless, bound to the release workflow's identity): + +```bash +cosign verify ghcr.io/artyomsv/spire-run-worker: \ + --certificate-identity-regexp '^https://github.com/artyomsv/code-spire/.github/workflows/release.yml@refs/tags/v' \ + --certificate-oidc-issuer https://token.actions.githubusercontent.com +``` | Image | What it is | Licence | Visibility | |---|---|---|---| @@ -233,8 +242,8 @@ to a registry only you can read; set `SPIRE_FACTORY_AGENT_IMAGE_CLAUDE_CODE` to `compose.ghcr.yml` names the unit images at `SPIRE_VERSION`, the run worker's own tag. **The run worker pulls a unit image only when its Docker daemon does not hold it**, so a moving tag is pulled once and never refreshed: after an upgrade on `edge`, units keep running the first `edge` that daemon -ever pulled, beside a newer run worker. Either pin `SPIRE_VERSION` to a `sha-` tag -(a version tag works only once `release.yml` publishes these images; a new tag is a new image, so it is pulled), or, on `edge`, pull the unit images on the run worker's +ever pulled, beside a newer run worker. Either pin `SPIRE_VERSION` to a release version or a `sha-` +tag (a new tag is a new image, so it is pulled), or, on `edge`, pull the unit images on the run worker's daemon at every upgrade: ```bash diff --git a/deploy/compose.ghcr.yml b/deploy/compose.ghcr.yml index 14c3e312..185ad457 100644 --- a/deploy/compose.ghcr.yml +++ b/deploy/compose.ghcr.yml @@ -203,8 +203,9 @@ services: # sandbox); this file names the published one, compose.yml the locally built one. # # The worker pulls a unit image only when its daemon does not hold it, so a moving tag is pulled - # once and never again. Pin SPIRE_VERSION to a sha- tag (a version tag works only once release.yml publishes these images); on `edge`, pull these - # three and the agent image by hand on every upgrade (deploy/README.md). + # once and never again. Pin SPIRE_VERSION to a release version (release.yml publishes every image here at it) + # or a sha- tag; on `edge`, pull these three and the agent image by hand on every upgrade + # (deploy/README.md). SPIRE_PUBLISHER_IMAGE: ${SPIRE_PUBLISHER_IMAGE:-ghcr.io/artyomsv/spire-publisher:${SPIRE_VERSION:-edge}} # Which run units hold their model credential in a model proxy (OFF, REVIEWS, ALL), and its image: the # proxied runs fail as IMAGE_UNAVAILABLE without the image, builds run on. See deploy/.env.example. diff --git a/docs/CICD-AND-PACKAGING.md b/docs/CICD-AND-PACKAGING.md index f9ba48d5..8b0d1089 100644 --- a/docs/CICD-AND-PACKAGING.md +++ b/docs/CICD-AND-PACKAGING.md @@ -300,10 +300,44 @@ entrypoint or a launcher builds its image before merge. Every apk-based image ge `compose.ghcr.yml` defaults the unit images to the published ones at `SPIRE_VERSION`. The run worker pulls a unit image only when its daemon lacks it (`DockerRunRuntime.ensureImage`), so a moving tag never updates on -its own: pin `sha-` (a version tag works for these images only once `release.yml` publishes them, a follow-up PR), or pull by hand on upgrade (`deploy/README.md`). +its own: pin a release version or `sha-`, or pull by hand on upgrade (`deploy/README.md`). -**Still open:** these images are amd64 only, unsigned and have no SBOM; that is `release.yml`'s work, with -the Kubernetes deployment of the factory, in their own PRs. +**Still open:** the Kubernetes deployment of the factory, in its own PR. The release half is §11. + +--- + +## 11. The factory images join release.yml (2026-11-15) + +[The design](superpowers/specs/2026-11-15-release-factory-images-design.md). `release.yml`'s `publish` matrix now +holds every image `docker.yml` publishes, ten in all, so a `v*` tag publishes each one as `:` and +`:latest` for `linux/amd64,linux/arm64`, with provenance and an SBOM, cosign-signed keyless by digest, after the +same `gate` round-trip. An operator can now pin `SPIRE_VERSION=` for every image `compose.ghcr.yml` +names; before, a version pinned the four reviewer images and left the factory on `sha-`. + +- **Same fields as `docker.yml`.** `apkUpgrade` sends `APK_UPGRADE_BUST=github.run_id` (the release sent none + before, so it restored the upgrade layer `docker.yml` had cached); `gradleDist` runs `installDist` for the + publisher and the model proxy first. +- **The agents' model catalogue.** A multi-arch push cannot run the image it builds, so the job first runs the + image's own build script (`build-codex.sh`, `build-claude-code.sh`) on the runner's default builder, reads + `dev.codespire.agent.models` back from that amd64 image, and passes it to the multi-arch build as + `AGENT_MODELS`. The trim logic stays in the script; the step runs before `setup-buildx` so the image is loaded. +- **arm64, checked statically.** `eclipse-temurin:25-jre-alpine` (already released for arm64 by the four + services), `alpine:3.20` and `node:22-bookworm-slim` are multi-arch; `@openai/codex@0.156.1` and + `@anthropic-ai/claude-code@2.1.319` each ship a `linux-arm64` optional dependency on npm. No arm64 image has + been run. +- **The gate.** Its `.env` lacked `SPIRE_OIDC_RUN_WORKER_SECRET`, which `compose.ghcr.yml` requires with `:?` + even for the profiled run worker (compose interpolates every service before filtering by profile), so `up` + would refuse the file; the last dry run, 2026-08-05, predates that requirement. The gate now also runs + `config -q` with `--profile factory`. It still starts only the reviewer stack. +- **Guards.** `ApkUpgradeIsNotCachedTest` reads both workflows' matrices, and + `DockerWorkflowBuildsEveryImageTest.theReleasePublishesEveryImageDockerYmlPublishes` fails when the two + publish different images or build one from different Dockerfiles. + +**Dry run.** Actions → release → Run workflow (or `gh workflow run release.yml --ref `). It runs the gate, +builds all ten images for both architectures without pushing, installs cosign, and creates and deletes a draft +release. It proves the gate's `.env` and the factory profile interpolate, every image builds on arm64 under +QEMU, and both catalogue reads produce a value. It does not prove signing or a push: a dry run signs and pushes +nothing. --- diff --git a/docs/HISTORY.md b/docs/HISTORY.md index b0e31204..1c3d9d4b 100644 --- a/docs/HISTORY.md +++ b/docs/HISTORY.md @@ -2462,3 +2462,22 @@ lives in `docs/`, the locked decisions in `docs/DECISIONS.md`, and claims no tes Anthropic asks to remove it, we comply." The image now carries the label, the anonymous-pull privacy check is gone, and no manual visibility step remains (ADR-032 amended 2026-11-15, LICENSING.md, deploy/README.md). Anthropic has not given permission; the removal path is to delete the package and drop it from the `agents` matrix. + +- **release.yml publishes the factory images at a version (2026-11-15, Part of #391).** A `v*` tag now publishes all + ten images docker.yml publishes, the run worker and the five run-unit images included, as `:` and + `:latest` for amd64 and arm64, with provenance and an SBOM, cosign-signed by digest, after the same gate + (docs/superpowers/specs/2026-11-15-release-factory-images-design.md, docs/CICD-AND-PACKAGING.md §11), so + `SPIRE_VERSION=` now pins every image compose.ghcr.yml names. The agent images read their model + catalogue by running their own build script on amd64 and passing the label on as `AGENT_MODELS`. Fixed on the + way: the release sent no `APK_UPGRADE_BUST`, and the gate's `.env` lacked `SPIRE_OIDC_RUN_WORKER_SECRET`, + which compose.ghcr.yml requires even for the profiled run worker (the last dry run, 2026-08-05, predates it); the + gate also renders the factory profile now. `ApkUpgradeIsNotCachedTest` reads both workflows, and + `DockerWorkflowBuildsEveryImageTest` fails when they publish different images. arm64 was checked statically + only (base images, and a `linux-arm64` npm package for both CLIs). Not run yet: a dry run or a tag with these + changes. + + Live proof of the GHCR images (2026-11-14/15, on the dev stack): spire-test #106 was reviewed (`RECONCILE` on a + Codex seat, approved, run `review-106-734b7396736b-3270b22e:1`) with `spire-agent-codex`, `spire-publisher`, + `spire-model-proxy` and `spire-egress-proxy` all pulled from GHCR at `sha-66dd409`. The `spire-run-worker` and + `spire-agent-claude-code` images and the packaged run worker's Docker socket on Linux (uid 1001) are not yet + live-proven (docs/UNVERIFIED.md). diff --git a/docs/UNVERIFIED.md b/docs/UNVERIFIED.md index 786e2177..30b716ec 100644 --- a/docs/UNVERIFIED.md +++ b/docs/UNVERIFIED.md @@ -136,7 +136,9 @@ subscription is refused `subscription_not_allowed` at every door. every path today, so the catch is defence in depth. - **No live run reached:** a re-review (`RECONCILE`), a reply (`FOLLOWUP`), a rate limit, an error result, a run killed before `result`, `ALL` mode, or a key whose kind is not its vendor's (`harness_credential.type` now picks the key). -- **The image is opt-in and not on GHCR** (`SPIRE_FACTORY_AGENT_IMAGE_CLAUDE_CODE`, default `none`). +- **The image is opt-in, and the published one has not run a review.** It is on GHCR (`:edge`, `:sha-`, and + `:` multi-arch from release.yml), but every live Claude Code review used a local build + (`SPIRE_FACTORY_AGENT_IMAGE_CLAUDE_CODE`, default `none`). Its arm64 build has never run. **What evidence would settle it.** A run's charge compared with its `total_cost_usd` and the console; a re-review and a reply on Claude Code; a rate-limited run; the canaries, the nested `CLAUDE.md` pair included, re-run on every CLI @@ -219,7 +221,13 @@ run as `CREDENTIAL_REJECTED` and marks the pool member; a refused path fails it - **Every image lookup failure reads as a missing image.** The worker's check before the unit is created reports any failure to describe or pull `SPIRE_RUN_CREDENTIAL_PROXY_IMAGE`, a daemon outage included, as `IMAGE_UNAVAILABLE`. -- **The image is not on GHCR**, and only the Docker arm builds the topology (there is no Kubernetes arm). +- **The published images are proven on one review, on amd64.** On 2026-11-14/15 the dev stack reviewed spire-test + #106 (RECONCILE on a Codex seat, approved, run `review-106-734b7396736b-3270b22e:1`) with `spire-agent-codex`, + `spire-publisher`, `spire-model-proxy` and `spire-egress-proxy` pulled from GHCR at `sha-66dd409`. Not + live-proven: the published `spire-run-worker` image, the packaged run worker's + access to the Docker socket on Linux as uid 1001, the published `spire-agent-claude-code`, any arm64 image, and a + versioned release (release.yml publishes these images since 2026-11-15; no tag has run it). Only the Docker arm + builds the topology (there is no Kubernetes arm). - **Address pool.** Each proxied unit takes two internal networks, so Docker's default address pool supports roughly half as many concurrent proxied units as unproxied ones; no test runs near the bound. From badc54cb795e84fe8502a70527ab570f34b08846 Mon Sep 17 00:00:00 2001 From: Artjoms Stukans Date: Sun, 15 Nov 2026 02:00:19 +0100 Subject: [PATCH 4/4] Record the release dry run and how to recover a failed tag Dry run 43657172281 built every image for amd64 and arm64 and passed the gate. A real tag that fails one image is recovered by re-running the failed jobs; a version is usable once its GitHub release exists. Part of #391 --- deploy/README.md | 3 ++- docs/CICD-AND-PACKAGING.md | 7 ++++++- docs/HISTORY.md | 5 +++-- docs/UNVERIFIED.md | 3 ++- 4 files changed, 13 insertions(+), 5 deletions(-) diff --git a/deploy/README.md b/deploy/README.md index ecb3186f..5a18e90a 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -214,7 +214,8 @@ cosign-signed by digest with an SBOM and provenance (`release.yml`). For anythin `SPIRE_VERSION` to a release version, or to a `sha-` between releases. Each image carries its licence as an OCI label and as a file (LICENSING.md, "Published images"). -To check a release image's signature (keyless, bound to the release workflow's identity): +To check a release image's signature (keyless, bound to the release workflow's identity). No real tag has published +the factory images yet (a dry run built them, 2026-11-15), so the first one is the first proof: ```bash cosign verify ghcr.io/artyomsv/spire-run-worker: \ diff --git a/docs/CICD-AND-PACKAGING.md b/docs/CICD-AND-PACKAGING.md index 8b0d1089..c13e8416 100644 --- a/docs/CICD-AND-PACKAGING.md +++ b/docs/CICD-AND-PACKAGING.md @@ -337,7 +337,12 @@ names; before, a version pinned the four reviewer images and left the factory on builds all ten images for both architectures without pushing, installs cosign, and creates and deletes a draft release. It proves the gate's `.env` and the factory profile interpolate, every image builds on arm64 under QEMU, and both catalogue reads produce a value. It does not prove signing or a push: a dry run signs and pushes -nothing. +nothing. Actions run 43657172281 on 2026-11-15 (a dry run on the PR branch, 45 minutes) succeeded. + +**A failed image on a real tag.** The matrix runs with `fail-fast: false`, and each job pushes `:` and +`:latest` as soon as its image is built. If one of the ten fails, the other nine are already published and their +`:latest` has moved, while `github-release` does not run. Recover by re-running the failed jobs of the same tag run: +the pushes are idempotent and the version is the tag. A version is usable only once its GitHub release exists. --- diff --git a/docs/HISTORY.md b/docs/HISTORY.md index 1c3d9d4b..f2d5d005 100644 --- a/docs/HISTORY.md +++ b/docs/HISTORY.md @@ -2473,8 +2473,9 @@ lives in `docs/`, the locked decisions in `docs/DECISIONS.md`, and claims no tes which compose.ghcr.yml requires even for the profiled run worker (the last dry run, 2026-08-05, predates it); the gate also renders the factory profile now. `ApkUpgradeIsNotCachedTest` reads both workflows, and `DockerWorkflowBuildsEveryImageTest` fails when they publish different images. arm64 was checked statically - only (base images, and a `linux-arm64` npm package for both CLIs). Not run yet: a dry run or a tag with these - changes. + only (base images, and a `linux-arm64` npm package for both CLIs). Actions run 43657172281 on 2026-11-15 (a dry run on the PR branch, 45 minutes) succeeded: the gate, all ten + images built for amd64 and arm64 under QEMU with both catalogue reads, and the draft release. Not run yet: a + real tag, so pushing, signing and the SBOM in GHCR are unproven. Live proof of the GHCR images (2026-11-14/15, on the dev stack): spire-test #106 was reviewed (`RECONCILE` on a Codex seat, approved, run `review-106-734b7396736b-3270b22e:1`) with `spire-agent-codex`, `spire-publisher`, diff --git a/docs/UNVERIFIED.md b/docs/UNVERIFIED.md index 30b716ec..d6fa2c11 100644 --- a/docs/UNVERIFIED.md +++ b/docs/UNVERIFIED.md @@ -226,7 +226,8 @@ run as `CREDENTIAL_REJECTED` and marks the pool member; a refused path fails it `spire-publisher`, `spire-model-proxy` and `spire-egress-proxy` pulled from GHCR at `sha-66dd409`. Not live-proven: the published `spire-run-worker` image, the packaged run worker's access to the Docker socket on Linux as uid 1001, the published `spire-agent-claude-code`, any arm64 image, and a - versioned release (release.yml publishes these images since 2026-11-15; no tag has run it). Only the Docker arm + versioned release (release.yml publishes these images since 2026-11-15; no tag has run it; the dry run 43657172281 built every image + for both architectures and passed the gate). Only the Docker arm builds the topology (there is no Kubernetes arm). - **Address pool.** Each proxied unit takes two internal networks, so Docker's default address pool supports roughly half as many concurrent proxied units as unproxied ones; no test runs near the bound.