Resolve the latest release image to a published version - #7786
Open
CharlieTLe wants to merge 2 commits into
Open
Resolve the latest release image to a published version#7786CharlieTLe wants to merge 2 commits into
CharlieTLe wants to merge 2 commits into
Conversation
integration/util.go derived the "latest release" image straight from the VERSION file. That holds on master, where VERSION is the last GA, but not on a release branch: VERSION is bumped to the version being prepared (e.g. 1.22.0-rc.0) long before the deploy job publishes that tag, and the integration job is a dependency of deploy. So the query fuzz leg would pull an image that does not exist yet. Resolve a pre-release version to the release preceding it instead, and add CORTEX_LATEST_RELEASE_IMAGE as an escape hatch for the cases the version math cannot cover (a major pre-release). The preload step in test-build-deploy.yml mirrors the same rule. Signed-off-by: Charlie Le <charlie_le@apple.com>
CharlieTLe
requested review from
a team and
danielblando
and removed request for
a team
August 20, 2026 19:09
Member
|
Thanks for catching this. I think the issue with this PR is: Can we ask the registry what is actually published? The registry is the only source of truth for "published".
|
Deriving the previous release from VERSION alone left one case broken, as SungJin1212 pointed out on cortexproject#7786: on the GA tag push VERSION is 1.22.0 with no pre-release suffix, so it resolves to v1.22.0 — an image that only `deploy` publishes, and `deploy` needs `integration` to pass first. Ask quay.io what actually exists instead. The registry is the only source of truth for "published", so list the GA tags (^v\d+\.\d+\.\d+$) and take the highest one that does not exceed VERSION, then export it as CORTEX_LATEST_RELEASE_IMAGE for the preload and test steps. The <= bound only changes the result when a newer release already exists on quay than the branch being tested, e.g. preparing 1.21.2 on release-1.21 after v1.22.0 has shipped. This also drops the bash mirror of the Go derivation, and stops guessing at the previous minor's .0: with the registry answering, 1.19.5-rc.0 resolves to the v1.19.1 that exists rather than a v1.19.4 that never shipped, and a major pre-release such as 2.0.0-rc.0 no longer needs a manual override. The derivation stays in integration/util.go as the offline fallback for local runs, where no env var is set and no network call is wanted. Signed-off-by: Charlie Le <charlie_le@apple.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
getLatestReleaseImage()inintegration/util.goderives the "latest release" imagestraight from the
VERSIONfile, and.github/workflows/test-build-deploy.ymlmirrors itas
docker pull quay.io/cortexproject/cortex:v$(cat testdata/VERSION).That holds on
master, whereVERSIONis the last GA and its image exists. It does nothold on a release branch:
VERSIONis bumped to the version being prepared (e.g.1.22.0-rc.0) long before anything publishes that tag, and thedeployjob that wouldpublish it has
needs: [build, test, lint, integration]. So theintegration_query_fuzzleg would try to pull an image that does not exist yet, and
release-1.22would be redfrom the moment
VERSIONgains its-rc.0suffix.The same applies on the GA tag push, where
VERSIONis1.22.0with no suffix: thev1.22.0image is only pushed bydeploy, which again runs afterintegration.This mechanism landed in #7737 (2026-07-30), after v1.21.1, so it has never been through
a release.
The fix
VERSIONnames the version being prepared, so it cannot answer "what is published".The registry can, so ask it — thanks @SungJin1212 for the suggestion.
A new
Resolve Latest Release Imagestep lists the GA tags (^v\d+\.\d+\.\d+$) publishedto quay.io, takes the highest one that does not exceed
VERSION, and exports it asCORTEX_LATEST_RELEASE_IMAGEfor the preload and test steps. It runs only on theintegration_query_fuzzmatrix leg, which is the only caller ofgetLatestReleaseImage().Against a registry whose newest GA is
v1.21.1:VERSION1.21.1v1.21.11.22.0-rc.0v1.21.11.22.0v1.21.11.21.2-rc.0v1.21.12.0.0-rc.0v1.21.11.19.0v1.19.01.19.5-rc.0v1.19.11.19.4The
<=bound (rather than simply "the newest published GA tag") only changes the resultwhen a newer release already exists on quay than the branch being tested, e.g. preparing
1.21.2onrelease-1.21afterv1.22.0has shipped.Setting the
CORTEX_LATEST_RELEASE_IMAGErepository variable bypasses the lookupentirely.
The bash mirror of the Go derivation is gone.
latestReleaseVersion()stays inintegration/util.goas the offline fallback for local runs, where nothing sets the envvar and a network call is unwelcome; CI always resolves against the registry.
Verification
bash -e(what Actionsactually uses — no
pipefail, nonounset) against the live registry, producing everyrow in the table above, plus errors for a
VERSIONbelow everything published and for amalformed
VERSION. The repository-variable override was exercised too.shellcheckis clean on the step.actionlintreports the same 13 pre-existing SC2086findings as
masterand no new ones.latestReleaseVersion()keeps its table test;getLatestReleaseImage()is tested end toend against a scratch checkout and for the env override.
go vetpasses with every integration build tag set.