Skip to content

fix: pin Arc chart image repositories to the resolved MCR path - #1379

Closed
Yoo Bin Shin (nibooy) wants to merge 1 commit into
mainfrom
arc-chart-image-repo
Closed

Yoo Bin Shin (nibooy) wants to merge 1 commit into
mainfrom
arc-chart-image-repo

Conversation

@nibooy

Copy link
Copy Markdown
Member

Description of your changes

The build-publish-mcr workflow resolves image tags from aks/v2/fleet but packages the chart from the plain release tag. When that tag predates a change to the chart's image repositories, the published chart pairs the new tags with the old path.

That is what happened to chart 0.18.16-3 (run). The run was dispatched from main, but publish-arc-helm-chart checked out v0.18.16 (23641d7). That commit is one behind #1373, so its values.yaml still pointed at mcr.microsoft.com/aks/fleet/.... The packaged chart referenced images like aks/fleet/refresh-token:v0.18.16-3, which exist only under aks/v2/fleet. Every Arc install hit ImagePullBackOff (68/68 clusters in centraluseuap), and EV2's AllRegionsFailureRate gate halted the rollout.

This PR adds two steps before packaging:

  • Point the chart at the resolved image repository. Overwrites each component's repository in values.yaml with mcr.microsoft.com/${MCR_FLEET_REPO}/<image>, so the repository always matches where the tags were resolved, whatever the release tag's values.yaml says. The step fails if a key cannot be rewritten. All five keys have been unchanged since at least v0.18.10, so this works on older release tags too.
  • Verify every chart image is published. Renders the chart with helm template and checks each referenced image against the MCR registry API, failing before anything is pushed if one is missing. This catches the whole class of chart/image mismatch, not just the repository move.

The image version variables move from the packaging step to the job env so both new steps can use them. The chart README now documents the override.

I have:

  • Run make reviewable to ensure this PR is ready for review. (Not run: the change touches only the workflow and README; actionlint and the chart tests were run instead.)

How has this code been tested

I extracted the two new steps from the workflow and ran them locally with the inputs of the failing run (MCR_FLEET_REPO=aks/v2/fleet, v0.18.16-3, v0.3.44-4):

Chart source Result
v0.18.16 (the chart that broke) Repositories rewritten to aks/v2/fleet; all 5 images found; exit 0
main No-op rewrite; all 5 images found; exit 0
v0.18.16, verify step only (no rewrite) All 5 aks/fleet/... images reported missing; exit 1, so the broken chart would not have been published

actionlint and go test ./charts/member-agent-arc/... pass.

Special notes for your reviewer

  • With this change, the repository values in values.yaml only matter when the chart is packaged outside the workflow. The workflow is now the source of truth for the published chart.
  • The verify step uses the same unauthenticated MCR registry API that prepare-variables already uses for tag resolution, so it adds no new credentials.
  • Chart 0.18.16-3 is already in the registry. Once this merges, re-running the workflow with releaseTag: v0.18.16 produces a corrected chart, but it needs a new image revision (-4) to get a distinct chart version.

🤖 Generated with Claude Code

The workflow resolves image tags from aks/v2/fleet but packages the chart from
the plain release tag. When that tag predates a change to the chart's image
repositories, the published chart pairs the new tags with the old path:
0.18.16-3 was packaged from v0.18.16, one commit before the move to
aks/v2/fleet, and referenced aks/fleet/*:v0.18.16-3, which does not exist.
Every install hit ImagePullBackOff and the canary rollout gate failed.

Overwrite each image repository with MCR_FLEET_REPO before packaging, so the
chart always points where the tags were resolved, regardless of the release
tag's values.yaml. Then render the chart and check every referenced image
against MCR, failing before publish if any is missing.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: yoobinshin <yoobinshin@microsoft.com>
@nibooy

Copy link
Copy Markdown
Member Author

(AI-generated) Closing: opened by mistake. PRs must come from a fork; #1378 is the PR of record. The arc-chart-image-repo branch on Azure/fleet is kept only for test-dispatching the workflow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant