Skip to content

Resolve the latest release image to a published version - #7786

Open
CharlieTLe wants to merge 2 commits into
cortexproject:masterfrom
CharlieTLe:release-1.22-pr-a-fix-latest-release-image
Open

Resolve the latest release image to a published version#7786
CharlieTLe wants to merge 2 commits into
cortexproject:masterfrom
CharlieTLe:release-1.22-pr-a-fix-latest-release-image

Conversation

@CharlieTLe

@CharlieTLe CharlieTLe commented Aug 20, 2026

Copy link
Copy Markdown
Member

What this fixes

getLatestReleaseImage() in integration/util.go derives the "latest release" image
straight from the VERSION file, and .github/workflows/test-build-deploy.yml mirrors it
as docker pull quay.io/cortexproject/cortex:v$(cat testdata/VERSION).

That holds on master, where VERSION is the last GA and its image exists. It does not
hold on a release branch: VERSION is bumped to the version being prepared (e.g.
1.22.0-rc.0) long before anything publishes that tag, and the deploy job that would
publish it has needs: [build, test, lint, integration]. So the integration_query_fuzz
leg would try to pull an image that does not exist yet, and release-1.22 would be red
from the moment VERSION gains its -rc.0 suffix.

The same applies on the GA tag push, where VERSION is 1.22.0 with no suffix: the
v1.22.0 image is only pushed by deploy, which again runs after integration.

This mechanism landed in #7737 (2026-07-30), after v1.21.1, so it has never been through
a release.

The fix

VERSION names the version being prepared, so it cannot answer "what is published".
The registry can, so ask it — thanks @SungJin1212 for the suggestion.

A new Resolve Latest Release Image step lists the GA tags (^v\d+\.\d+\.\d+$) published
to quay.io, takes the highest one that does not exceed VERSION, and exports it as
CORTEX_LATEST_RELEASE_IMAGE for the preload and test steps. It runs only on the
integration_query_fuzz matrix leg, which is the only caller of getLatestReleaseImage().

Against a registry whose newest GA is v1.21.1:

VERSION resolves to
1.21.1 v1.21.1 master's steady state
1.22.0-rc.0 v1.21.1 release branch preparing a minor
1.22.0 v1.21.1 the GA tag build
1.21.2-rc.0 v1.21.1 release branch preparing a patch
2.0.0-rc.0 v1.21.1 a major pre-release
1.19.0 v1.19.0 an older branch — note it does not jump forward
1.19.5-rc.0 v1.19.1 the newest published patch, not a guessed 1.19.4

The <= bound (rather than simply "the newest published GA tag") only changes the result
when a newer release already exists on quay than the branch being tested, e.g. preparing
1.21.2 on release-1.21 after v1.22.0 has shipped.

Setting the CORTEX_LATEST_RELEASE_IMAGE repository variable bypasses the lookup
entirely.

The bash mirror of the Go derivation is gone. latestReleaseVersion() stays in
integration/util.go as the offline fallback for local runs, where nothing sets the env
var and a network call is unwelcome; CI always resolves against the registry.

Verification

  • The workflow step was extracted from the YAML and run under bash -e (what Actions
    actually uses — no pipefail, no nounset) against the live registry, producing every
    row in the table above, plus errors for a VERSION below everything published and for a
    malformed VERSION. The repository-variable override was exercised too.
  • shellcheck is clean on the step. actionlint reports the same 13 pre-existing SC2086
    findings as master and no new ones.
  • latestReleaseVersion() keeps its table test; getLatestReleaseImage() is tested end to
    end against a scratch checkout and for the env override.
  • go vet passes with every integration build tag set.

integration/util.go derived the "latest release" image straight from the VERSION
file. That holds on master, where VERSION is the last GA, but not on a release
branch: VERSION is bumped to the version being prepared (e.g. 1.22.0-rc.0) long
before the deploy job publishes that tag, and the integration job is a dependency
of deploy. So the query fuzz leg would pull an image that does not exist yet.

Resolve a pre-release version to the release preceding it instead, and add
CORTEX_LATEST_RELEASE_IMAGE as an escape hatch for the cases the version math
cannot cover (a major pre-release). The preload step in test-build-deploy.yml
mirrors the same rule.

Signed-off-by: Charlie Le <charlie_le@apple.com>
@dosubot dosubot Bot added ci/cd go Pull requests that update Go code labels Aug 20, 2026
@CharlieTLe
CharlieTLe requested review from a team and danielblando and removed request for a team August 20, 2026 19:09
@SungJin1212

SungJin1212 commented Sep 7, 2026

Copy link
Copy Markdown
Member

Thanks for catching this.

I think the issue with this PR is:
When we push the v1.22.0 tag, VERSION is 1.22.0, and this resolves to quay.io/cortexproject/cortex:v1.22.0. That image is published by deploy, and deploy needs integration to pass.

Can we ask the registry what is actually published? The registry is the only source of truth for "published".

  1. Pick the highest published GA tag (^v\d+\.\d+\.\d+$) that is <= the base of VERSION
    ㄴ The <= bound only changes the result when a newer release already exists on quay than the branch being tested (e.g. preparing 1.21.2 on release-1.21 after v1.22.0 shipped).
  2. Export it as CORTEX_LATEST_RELEASE_IMAGE

Deriving the previous release from VERSION alone left one case broken, as
SungJin1212 pointed out on cortexproject#7786: on the GA tag push VERSION is 1.22.0 with no
pre-release suffix, so it resolves to v1.22.0 — an image that only `deploy`
publishes, and `deploy` needs `integration` to pass first.

Ask quay.io what actually exists instead. The registry is the only source of
truth for "published", so list the GA tags (^v\d+\.\d+\.\d+$) and take the
highest one that does not exceed VERSION, then export it as
CORTEX_LATEST_RELEASE_IMAGE for the preload and test steps.

The <= bound only changes the result when a newer release already exists on
quay than the branch being tested, e.g. preparing 1.21.2 on release-1.21 after
v1.22.0 has shipped.

This also drops the bash mirror of the Go derivation, and stops guessing at the
previous minor's .0: with the registry answering, 1.19.5-rc.0 resolves to the
v1.19.1 that exists rather than a v1.19.4 that never shipped, and a major
pre-release such as 2.0.0-rc.0 no longer needs a manual override.

The derivation stays in integration/util.go as the offline fallback for local
runs, where no env var is set and no network call is wanted.

Signed-off-by: Charlie Le <charlie_le@apple.com>
@CharlieTLe
CharlieTLe requested a review from a team as a code owner September 8, 2026 18:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/cd go Pull requests that update Go code size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants