Repository navigation
fix: pin Arc chart image repositories to the resolved MCR path - #1379
Closed
Yoo Bin Shin (nibooy) wants to merge 1 commit into
Closed
Yoo Bin Shin (nibooy) wants to merge 1 commit into
Yoo Bin Shin (nibooy) wants to merge 1 commit into
Conversation
The workflow resolves image tags from aks/v2/fleet but packages the chart from the plain release tag. When that tag predates a change to the chart's image repositories, the published chart pairs the new tags with the old path: 0.18.16-3 was packaged from v0.18.16, one commit before the move to aks/v2/fleet, and referenced aks/fleet/*:v0.18.16-3, which does not exist. Every install hit ImagePullBackOff and the canary rollout gate failed. Overwrite each image repository with MCR_FLEET_REPO before packaging, so the chart always points where the tags were resolved, regardless of the release tag's values.yaml. Then render the chart and check every referenced image against MCR, failing before publish if any is missing. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: yoobinshin <yoobinshin@microsoft.com>
Member
Author
|
(AI-generated) Closing: opened by mistake. PRs must come from a fork; #1378 is the PR of record. The |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of your changes
The
build-publish-mcrworkflow resolves image tags fromaks/v2/fleetbut packages the chart from the plain release tag. When that tag predates a change to the chart's image repositories, the published chart pairs the new tags with the old path.That is what happened to chart
0.18.16-3(run). The run was dispatched frommain, butpublish-arc-helm-chartchecked outv0.18.16(23641d7). That commit is one behind #1373, so itsvalues.yamlstill pointed atmcr.microsoft.com/aks/fleet/.... The packaged chart referenced images likeaks/fleet/refresh-token:v0.18.16-3, which exist only underaks/v2/fleet. Every Arc install hitImagePullBackOff(68/68 clusters in centraluseuap), and EV2'sAllRegionsFailureRategate halted the rollout.This PR adds two steps before packaging:
repositoryinvalues.yamlwithmcr.microsoft.com/${MCR_FLEET_REPO}/<image>, so the repository always matches where the tags were resolved, whatever the release tag'svalues.yamlsays. The step fails if a key cannot be rewritten. All five keys have been unchanged since at leastv0.18.10, so this works on older release tags too.helm templateand checks each referenced image against the MCR registry API, failing before anything is pushed if one is missing. This catches the whole class of chart/image mismatch, not just the repository move.The image version variables move from the packaging step to the job
envso both new steps can use them. The chart README now documents the override.I have:
make reviewableto ensure this PR is ready for review. (Not run: the change touches only the workflow and README;actionlintand the chart tests were run instead.)How has this code been tested
I extracted the two new steps from the workflow and ran them locally with the inputs of the failing run (
MCR_FLEET_REPO=aks/v2/fleet,v0.18.16-3,v0.3.44-4):v0.18.16(the chart that broke)aks/v2/fleet; all 5 images found; exit 0mainv0.18.16, verify step only (no rewrite)aks/fleet/...images reported missing; exit 1, so the broken chart would not have been publishedactionlintandgo test ./charts/member-agent-arc/...pass.Special notes for your reviewer
repositoryvalues invalues.yamlonly matter when the chart is packaged outside the workflow. The workflow is now the source of truth for the published chart.prepare-variablesalready uses for tag resolution, so it adds no new credentials.0.18.16-3is already in the registry. Once this merges, re-running the workflow withreleaseTag: v0.18.16produces a corrected chart, but it needs a new image revision (-4) to get a distinct chart version.🤖 Generated with Claude Code