Repository navigation
CAMEL-26492: check multi-arch manifest before proposing container upgrades - #29149
Conversation
…rades The container version checker proposed upgrades even when the new image tag lacked linux/s390x or linux/ppc64le support, breaking CI on those architectures. Changes: - Add a module-level REQUIRED_ARCHITECTURES constant ['linux/s390x', 'linux/ppc64le'] - Add ContainerRegistryAPI.check_architectures(image, tag) that calls the Docker Registry v2 manifest list endpoint (GET /v2/<repo>/manifests/<tag> with Accept: application/vnd.docker.distribution.manifest.list.v2+json) and returns the required architectures absent from the manifest. - DockerHubAPI overrides check_architectures to obtain an anonymous Bearer token from auth.docker.io before fetching the manifest. - QuayAPI overrides check_architectures to use the quay.io/v2/... endpoint. - ContainerImage gains a has_arch_override flag that is set when the properties file already declares an arch-specific sibling (e.g. keycloak.container.ppc64le). Such images are exempt from the check. - VersionCheckResult gains an arch_missing field. When the top candidate is missing required architectures, newer_versions is left empty so no PR is opened for that image; arch_missing is populated for reporting. - print_results shows a dedicated 'ARCH-INCOMPLETE IMAGES' section. - output_json includes arch_missing in the JSON payload. - Add --skip-arch-check CLI flag to bypass the verification. - GHA workflow step summary reports the number of arch-blocked images. - Add test_check_container_versions.py with 19 unit tests covering all new code paths.
|
🌟 Thank you for your contribution to the Apache Camel project! 🌟 🐫 Apache Camel Committers, please review the following items:
|
There was a problem hiding this comment.
Also need to adapt the multi-arch check based on if the related tests are skipped on s390x and ppc64le. or as a first iteration, to create the PR even if the multi-arch is not available but providing a big warning to let end-users check if there is the need for these architectures or not.
EDIT: so checking in dependent modules if skipITs.ppc64le and skipITs.s390x is set to true
✅ Local unit-test results for
|
davsclaus
left a comment
There was a problem hiding this comment.
Thanks for picking this up; checking the manifest list before proposing an upgrade is the right place for it. Adding to @apupier's review:
(1) The CI failure comes from this PR: pr-ci-scripts-validation.yml runs the unittest discovery without installing requirements.txt, so importing the script fails on colorama (check-container-versions.py:112). Installing the requirements there, as check-container-versions.yml:57 does, should fix it.
(2) The check fails open in exactly the cases it is meant to catch. A 200 without manifests (lines 348/394/465) is a single-arch image: for example, mcr.microsoft.com mssql/server answers with a schema1 manifest for this Accept header. A non-200 (342/390/461) is what ghcr.io and docker.elastic.co return without a token, and test-infra uses both. Could a single manifest count as "required archs missing", the 401 Bearer handshake be done, and a truly unverifiable result still open the PR but with a visible "multi-arch not verified" warning?
(3) The check verifies newer_versions[0] (line 1070), but the workflow bumps to latest_version (check-container-versions.yml:142/161). That is sorted_versions[0] (line 1056) and can be a pre-release that newer_versions filtered out, so the tag checked may not be the tag proposed.
(4) has_arch_override (lines 893-906, 940) exempts every required arch when any arch sibling exists, and most overrides are .ppc64le only (keycloak, mongodb, neo4j, pulsar, elasticsearch...), so s390x is never checked for them. Subtracting only the overridden archs, together with the skipITs.<arch> point from @apupier, would make the exemption precise.
Small things: media_type (line 346) is unused, and the three copies of the manifest parsing could share one helper.
Claude Code on behalf of Claus Ibsen (davsclaus). This review was generated by an AI agent and may contain inaccuracies; please verify all suggestions before applying.
oscerd
left a comment
There was a problem hiding this comment.
Solid, and the fail-closed posture is exactly right for this. When latest_version (the tag the workflow actually bumps to) is missing a required arch, clearing newer_versions and setting is_latest=True means the bot simply does not propose the upgrade, rather than proposing one that would break s390x/ppc64le CI. The exemption set is also correct: the union of arch_overrides and skipped_arches is subtracted before the check, so an arch that has an explicit *.ppc64le property sibling or skipITs.ppc64le=true in the nearest pom is not treated as missing. Targeting sorted_versions[0] instead of newer_versions[0] is the right call too, since a filtered-out pre-release should not be what you verify.
Test coverage is the strong point here: CheckImageVersionsArchGatingTest exercises each decision directly (blocked-when-missing, exempt-on-override, exempt-on-skipITs, targets-latest-not-newer, the --skip-arch-check bypass, and the JSON field), and the Bearer-challenge and pom-reading helpers each get their own tests. The 401 to Bearer re-auth for ghcr.io / docker.elastic.co is a nice robustness touch.
One non-blocking note: the conservatism is whole-list, so if sorted_versions[0] fails the arch check but an older entry in newer_versions is actually arch-complete, that older upgrade is skipped too. The comment calls this out, and skipping a real upgrade is clearly the safe side to err on, so no change needed. LGTM.
Claude Code on behalf of oscerd
davsclaus
left a comment
There was a problem hiding this comment.
Thanks for the update. (1), (3) and the per-arch override exemption from (4) are fixed, and the Bearer handshake works: docker.elastic.co and ghcr.io both give an anonymous token and then the index. Still open:
- mirror.gcr.io images are never checked (check-container-versions.py:314-315, used for the manifest URL).
normalize_registry_urlturnsmirror.gcr.iointogcr.io, andhttps://gcr.io/v2/apache/kafka/manifests/3.9.1answers 401; the challenge gives an empty anonymous token and the retry is 401 again, so the check passes.https://mirror.gcr.io/v2/apache/kafka/manifests/3.9.1answers 200 anonymously with the index. That covers most of the test-infra images. Could we usemirror.gcr.ioas is for manifests, and add a test for it? - Single-arch is still "nothing missing" (lines 423/455/515;
test_returns_empty_on_single_arch_manifestasserts it), and there is no "multi-arch not verified" warning when the check cannot run. A manifest withoutmanifests(or schema1 with a top-levelarchitecture) means one arch only, so it can count as missing. skipITsis read from the test-infra module pom (_read_skip_its_from_pom, :911). Only 1 of the 58 poms that setskipITs.ppc64le/skipITs.s390xis a test-infra pom; the rest are the component poms @apupier mentioned. With the new fail-closed path, elasticsearch (9.1.0 is amd64/arm64 only; both dependent components skip s390x) and spire-server/agent (camel-spiffe skips both arches) would never be proposed again. Options: read the poms that depend oncamel-test-infra-<x>; only block when the new tag drops an arch the current tag has; or open the PR with a prominent warning, as @apupier suggested first.- Minor: the
*.ppc64leoverride properties are checked as images themselves, so they would need s390x. They pass today only because icr.io answers schema1; skipping the check for keys that are themselves arch overrides would avoid that. Nit:import re as _reat :385/:936 duplicates the top-levelimport re.
Claude Code on behalf of davsclaus. This review was generated by an AI agent and may contain inaccuracies; please verify all suggestions before applying.
…ecker - Fix mirror.gcr.io manifest checks: add GCRAPI.check_architectures override that uses the image's original registry (mirror.gcr.io) instead of normalising to gcr.io, which returns 401 for public images - Fix single-arch manifests: a response with no 'manifests' key (schema1 or single-arch) now correctly reports all REQUIRED_ARCHITECTURES as missing, rather than silently allowing the upgrade - Detect skipITs.<arch> from component poms: add _build_component_skip_its_map that pre-scans all pom.xml files in the scan path at startup, building a map from camel-test-infra-X artifact IDs to their consumer component's skipped arches. parse_properties_file merges these with the local pom walk-up result. - Skip arch-override sibling keys (e.g. keycloak.container.ppc64le) from being parsed as standalone images; they are arch-specific registry references, not independent images to be version-checked - Remove duplicate 'import re as _re' inside functions; use the top-level 're' - Tests: rename test_returns_empty_on_single_arch_manifest to test_returns_missing_arches_on_single_arch_manifest and update assertion; add GCRAPICheckArchitecturesTest (mirror.gcr.io, single-arch); add ComponentSkipItsMapTest (3 cases for _build_component_skip_its_map) Total: 28 → 33 unit tests
davsclaus
left a comment
There was a problem hiding this comment.
Thanks for the update. Addressed since my last review:
- mirror.gcr.io:
GCRAPI.check_architecturesusesimage.registryas is, with a test. - Single-arch manifests: a payload without
manifestsnow counts as missing in all fourcheck_architecturesimplementations. - Arch-override keys: keys like
keycloak.container.ppc64leare no longer checked as images. - The
import re as _reduplicates are gone.
Still open:
-
The component
skipITsmap is empty in CI.run_checkbuilds it with_build_component_skip_its_map(scan_path)(check-container-versions.py:1344), which only searches belowscan_path, and the workflow scanstest-infraby default (check-container-versions.yml:62/74). Of the poms that setskipITs.ppc64le/skipITs.s390xto true, nearly all are component poms (e.g.components/camel-elasticsearch*/pom.xml,components/camel-spiffe/pom.xml), so elasticsearch and spire are still never proposed.ComponentSkipItsMapTestpasses because it writescomponents/...inside its own temporary scan directory. Could the map be built from the repository root (walk up fromscan_pathto the directory containingcomponents/, or a--repo-rootoption)? A test runningrun_checkwithscan_path=<root>/test-infrawould cover it. -
Union across consumers exempts too much once (1) works.
_build_component_skip_its_mapcollects every arch that any consumer pom skips (test_merges_arches_from_multiple_pomsasserts that). A pom-levelskipITsusually exists because of one of the module's services, not everycamel-test-infra-*it depends on. E.g.camel-test-infra-postgreswould lose s390x because ofcamel-pgvector, althoughcamel-sql,camel-jtaandcamel-pg-replication-slotrun there;camel-test-infra-artemiswould lose s390x because ofcamel-activemq/camel-paho-mqtt5, whilecamel-jms,camel-sjms,camel-amqpstill run there. A tag that drops s390x would then be proposed and break those ITs. Exempting an arch only when every consumer skips it (an intersection) keeps the elasticsearch and spiffe cases working without this gap. -
The "not verified" case is still silent. When the check cannot run (non-200 after the token retry, network error, unknown format),
check_architecturesreturns[]and the PR opens with no warning. A short "multi-arch not verified" line in the PR body or step summary would be enough.
Claude Code on behalf of Claus Ibsen (davsclaus). This review was generated by an AI agent and may contain inaccuracies; please verify all suggestions before applying.
…epo-root scan, unverified warning) - Fix Issue 1: Build the component skipITs map from the repository root (not just scan_path) so that component poms under components/ are found even when scanning test-infra/. Added _find_repo_root() helper that walks up to the directory containing components/ or .git. Also filter out test-infra module poms that declare their own artifactId (not consumers). - Fix Issue 2: Change _build_component_skip_its_map to use intersection instead of union across consumer modules. An arch is now only exempted when EVERY consumer pom that references a test-infra module skips it. This prevents camel-test-infra-postgres from losing s390x just because camel-pgvector skips it while camel-sql, camel-jta etc. still need it. - Fix Issue 3: Add arch_unverified field to VersionCheckResult. When check_architectures cannot verify (network error, auth failure, unknown format), it now returns None instead of []. The upgrade is still proposed but arch_unverified=true is set, which triggers a warning in the step summary and human-readable output. Also added to JSON output. Co-authored-by: Claude <noreply@anthropic.com>
✅ Review issues addressed in
|
davsclaus
left a comment
There was a problem hiding this comment.
Thanks for the update. Addressed since my last review:
- Unverified multi-arch:
check_architecturesreturnsNone,arch_unverifiedis set, and the step summary says so (check-container-versions.yml:412-416). - Union vs intersection: an arch is now only exempt when every consumer skips it (check-container-versions.py:1045-1058).
- Repo root: the skipITs map is built from the repository root (
_find_repo_root, :1368/:1406).
Still open:
camel-test-infra-allcancels every exemption.test-infra/camel-test-infra-all/pom.xmldeclares everycamel-test-infra-*module (121 artifactId references) and has noskipITs. So with the intersection it vetoes all of them: the dir-name filter at :1027-1030 only drops a pom's own artifactId. Running_build_component_skip_its_mapagainst the current tree gives one entry,camel-test-infra-duckdb, the only module not in-all. With the poms undertest-infra/left out, 37 modules get exemptions, including elasticsearch[linux/s390x]and spiffe[linux/ppc64le, linux/s390x]. As it is, elasticsearch is still never proposed: 9.1.0/9.2.0 on docker.elastic.co are amd64/arm64 only, and.ppc64leonly covers ppc64le. Could we skip consumer poms undertest-infra/(or whose directory starts withcamel-test-infra-), and add an-allpom to theRunCheckRepoRootTestfixture?- Minor:
RunCheckRepoRootTest(test_check_container_versions.py:641) reaches docker.elastic.co throughrun_check->check_image_versions, and the assertions underif results:/if es_results:(:679) may not run. Patchingcheck_image_versionswould keep the test offline and make the assertions unconditional. - Nit: the three new test classes (:597, :628, :685) come after
if __name__ == "__main__": unittest.main().discoverstill finds them, but running the file directly skips them, so moving the guard to the end would fix it. - The PR description still says the union-based map scans only the scan path. Could it mention the intersection, the repo-root scan and
arch_unverified?
Claude Code on behalf of davsclaus. This review was generated by an AI agent and may contain inaccuracies; please verify all suggestions before applying.
…emption, offline tests, main guard)
davsclaus
left a comment
There was a problem hiding this comment.
Thanks, all four points from my last review are fixed in 97216b7:
camel-test-infra-all:_build_component_skip_its_mapnow skips poms incamel-test-infra-*directories (check-container-versions.py:1012-1017), andtest_test_infra_all_pom_is_ignoredcovers it. Outside those directories, no pom references more than sixcamel-test-infra-*artifacts (camel-langchain4j-embeddingsis the largest), so the remaining consumers are real component poms. The elasticsearch and spiffe exemptions now work.RunCheckRepoRootTest:check_image_versionsis patched, so the test is offline and its assertions always run.- The
__main__guard is at the end of the file. - The description covers the intersection, the repo-root scan and
arch_unverified.
Two nits, not blocking:
- The description says
RunCheckRepoRootTestchecks thatcamel-test-infra-alldoes not cancel exemptions, but its fixture has no-allpom:ComponentSkipItsMapTest.test_test_infra_all_pom_is_ignoredis the test for that. Also, the finalassertIn("linux/s390x", es_result.image.skipped_arches)checks the mocked result, so it can't fail. The_component_skip_its_mapassertions above it are the ones that matter. pom_file.partsis the absolute path, so a checkout under a directory namedcamel-test-infra-...would skip every pom.pom_file.relative_to(base).partswould rule that out.
@apupier, your skipITs point is in now (component poms, intersection across consumers), in case you want to take another look at your change request.
Reviewed with Claude Code on behalf of davsclaus. This review was generated by an AI agent and may contain inaccuracies; please verify all suggestions before applying.
davsclaus
left a comment
There was a problem hiding this comment.
Re-review of the new commit 7de7084. Matching the exclude dirs and the camel-test-infra-* prefix against pom_file.relative_to(base).parts fixes the false positives from a checkout path that contains target, build or camel-test-infra-... above the scan root. relative_to(base) cannot fail here, because rglob only yields paths under the resolved base. LGTM.
Reviewed with Claude Code on behalf of davsclaus. This review was generated by an AI agent and may contain inaccuracies; please verify all suggestions before applying.
…al assertion
The class docstring now says "via _find_repo_root" to match what it actually
tests (repo-root discovery when scan_path is test-infra). The description in
davsclaus's review pointed out it claimed to cover the camel-test-infra-all
exemption, which is covered by ComponentSkipItsMapTest.test_test_infra_all_pom_is_ignored.
The final assertIn("linux/s390x", es_result.image.skipped_arches) only
verified the mocked return value (which always has s390x in skipped_arches),
so it could never catch a regression. The _component_skip_its_map assertions
above it are the meaningful checks.
davsclaus
left a comment
There was a problem hiding this comment.
Re-review of 216484d. The new commit only touches test_check_container_versions.py. It removes the assertion on the mocked result that could not fail, and the RunCheckRepoRootTest docstring now names _find_repo_root. The _component_skip_its_map assertions that cover the repo-root scan stay, and the description now credits the camel-test-infra-all exclusion to ComponentSkipItsMapTest. That closes the last nits from my earlier review. No workflow change in this commit. LGTM.
Claude Code on behalf of davsclaus. This review was generated by an AI agent and may contain inaccuracies; please verify all suggestions before applying.
|
@apupier Your concern about adapting the arch check based on Davsclaus has reviewed and approved three times. Could you take another look and dismiss your change request if you're satisfied? |
Summary
The container version checker was proposing upgrades even when the new image tag lacked
linux/s390xorlinux/ppc64lesupport. This broke CI on those architectures.Changes
check-container-versions.pyREQUIRED_ARCHITECTURES = ['linux/s390x', 'linux/ppc64le']ContainerRegistryAPI._parse_manifest_listand_fetch_manifesthelpers (removes three copies of identical manifest-parsing logic; also handles 401 → Bearer challenge re-authentication automatically for registries likeghcr.ioanddocker.elastic.cothat require a token even for public images)ContainerRegistryAPI._bearer_token_from_challengeto parseWww-AuthenticateBearer challenges and exchange them for anonymous tokensDockerHubAPI.check_architecturesandQuayAPI.check_architecturesnow delegate to the shared helpers instead of duplicating the manifest-fetch logicGCRAPI.check_architecturesoverride that uses the image's original registry URL (e.g.mirror.gcr.io) instead of normalising togcr.io:mirror.gcr.ioanswers manifest requests anonymously (200) whilegcr.ioreturns 401 for public images, causing the check to pass silentlymanifestskey in the response, or a schema1 manifest) now count as arch-missing and block the upgrade, rather than silently allowing itlatest_version(=sorted_versions[0], the version the workflow actually bumps to) instead ofnewer_versions[0], which could be a pre-release filtered out before the bumpContainerImagegainsarch_overrides: List[str](the specificlinux/<arch>entries that have explicit arch-specific property siblings) andskipped_arches: List[str](arches withskipITs.<arch>=truein the nearest module pom.xml); the arch check is now per-arch precise: only arches not covered by either list are checked against the manifestContainerVersionChecker._read_skip_its_from_pomthat walks up the directory tree from eachcontainer.propertiesfile, finds the nearestpom.xml, and readsskipITs.ppc64le/skipITs.s390xpropertiesContainerVersionChecker._build_component_skip_its_mapthat pre-scans allpom.xmlfiles from the repository root at startup, building a map fromcamel-test-infra-Xartifact IDs to the intersection of skipped arches across all consumer component poms (e.g.components/camel-elasticsearch-rest-client/pom.xml). An arch is only exempt when every consumer skips it (intersection, not union). Poms undertest-infra/camel-test-infra-*directories are excluded from the scan — they are test-infra module poms, not consumer poms, and aggregator poms likecamel-test-infra-allwould otherwise veto all exemptions.parse_properties_filemerges these with the local pom walk-up result, so images whoseskipITsonly appear in the component pom (not the test-infra module pom) are correctly exemptedkeycloak.container.ppc64le) are no longer parsed as standalone images and subjected to their own version/arch check; they are arch-specific registry references, not independent imagesimport re as _reinside helper functions; use the top-levelreimport throughoutVersionCheckResultgains anarch_missingfield. When the top candidate lacks required architectures,newer_versionsis cleared so no PR is opened, andarch_missingis populated for reporting. When the arch check cannot be verified (e.g. authentication fails and the registry returns no index),arch_unverifiedis set and the step summary reports the image as unverified rather than silently passingprint_resultsadds a dedicated «ARCH-INCOMPLETE IMAGES» sectionoutput_jsonincludesarch_missingandarch_unverifiedin the JSON payload--skip-arch-checkCLI flag to bypass the verificationcheck-container-versions.ymlpr-ci-scripts-validation.ymlpip install -r requirements.txtstep beforepython3 -m unittest discoversocolorama(and other deps) are available when the tests import the main scripttest_check_container_versions.py(updated: 19 → 44 unit tests)test_arch_check_targets_latest_version_not_just_newer,test_partial_arch_override_still_checks_unoverridden_arch,test_skipped_arches_exempted_from_checktest_returns_empty_on_single_arch_manifestrenamed totest_returns_missing_arches_on_single_arch_manifestand assertion updated to expect the required arches as missingReadSkipItsPomTest(3 tests) for_read_skip_its_from_pomBearerTokenChallengeTest(3 tests) for_bearer_token_from_challengeGCRAPICheckArchitecturesTest(2 tests) forGCRAPI.check_architectures: verifies thatmirror.gcr.iois used as-is in the manifest URL, and that single-arch manifests are blockedComponentSkipItsMapTest(4 tests) for_build_component_skip_its_map: verifies discovery of skipped arches from component poms, correct handling of no-skipITs cases, merging across multiple poms, and thatcamel-test-infra-allaggregator poms are excluded from the intersectionRunCheckRepoRootTest(offline, fully mocked): verifies that_build_component_skip_its_mapis built from the repo root (not the scan path), so component poms undercomponents/are found even when scanningtest-infra/;check_image_versionsis patched so the test runs offline with unconditional assertions