Repository navigation
Open
Labels
bugIssue/PR to expose/discuss/fix a bugIssue/PR to expose/discuss/fix a bug
Description
Evidence
1. Manifest-list media type per version
$ for tag in v25.10.0 v26.3.3 v26.7.0 v26.7.1 v26.7.2; do
echo "=== $tag ==="
skopeo inspect --raw docker://nvcr.io/nvidia/gpu-operator:$tag | jq -r .mediaType
done
=== v25.10.0 ===
application/vnd.docker.distribution.manifest.list.v2+json
=== v26.3.3 ===
application/vnd.docker.distribution.manifest.list.v2+json
=== v26.7.0 ===
application/vnd.docker.distribution.manifest.list.v2+json
=== v26.7.1 ===
application/vnd.oci.image.index.v1+json
=== v26.7.2 ===
application/vnd.oci.image.index.v1+json
2. Layer media types per version (amd64)
$ for tag in v25.10.0 v26.3.3 v26.7.0 v26.7.1 v26.7.2; do
echo "=== $tag ==="
digest=$(skopeo inspect --raw docker://nvcr.io/nvidia/gpu-operator:$tag \
| jq -r '.manifests[] | select(.platform.architecture=="amd64") | .digest')
skopeo inspect --raw docker://nvcr.io/nvidia/gpu-operator@$digest \
| jq -r '.layers[].mediaType' | sort | uniq -c
done
=== v25.10.0 ===
36 application/vnd.docker.image.rootfs.diff.tar.gzip
=== v26.3.3 ===
42 application/vnd.docker.image.rootfs.diff.tar.gzip
=== v26.7.0 ===
1 application/vnd.docker.image.rootfs.diff.tar
42 application/vnd.docker.image.rootfs.diff.tar.gzip
=== v26.7.1 ===
1 application/vnd.oci.image.layer.v1.tar
42 application/vnd.oci.image.layer.v1.tar+gzip
=== v26.7.2 ===
1 application/vnd.oci.image.layer.v1.tar
42 application/vnd.oci.image.layer.v1.tar+gzip
3. The uncompressed layer is confirmed at the byte level, not just a label
$ TOKEN=$(curl -s "https://nvcr.io/token?service=nvcr.io&scope=repository:nvidia/gpu-operator:pull" \
| jq -r .token)
$ amd64_digest=$(skopeo inspect --raw docker://nvcr.io/nvidia/gpu-operator:v26.7.2 \
| jq -r '.manifests[] | select(.platform.architecture=="amd64") | .digest')
$ layer_digest=$(skopeo inspect --raw docker://nvcr.io/nvidia/gpu-operator@$amd64_digest \
| jq -r '.layers[] | select(.mediaType | contains("gzip") | not) | .digest')
$ curl -sL -H "Authorization: Bearer $TOKEN" \
"https://nvcr.io/v2/nvidia/gpu-operator/blobs/$layer_digest" -o layer.bin
$ file -b layer.bin
POSIX tar archive (GNU)
$ tar -tvf layer.bin
etc/
etc/ssl/
etc/ssl/fipsmodule.cnf
etc/ssl/openssl-fips.cnf
etc/ssl/openssl-nofips.cnf
This same layer (containing OpenSSL/FIPS config files) is present,
uncompressed, in v26.7.0, v26.7.1, and v26.7.2. It does not exist at all
in v25.10.0 or v26.3.3.
Impact
Customers mirroring certified operator bundles (e.g. via oc-mirror) to
a disconnected Red Hat Quay registry fail with:
[ERROR] : [Worker] error mirroring image
docker://nvcr.io/nvidia/cloud-native/<image>@sha256:...
error: Manifest list must be converted to type
"application/vnd.docker.distribution.manifest.list.v2+json" ...
"Instructed to preserve digests"
The only workaround today is a manual skopeo copy --format v2s2 --multi-arch=all re-push per image, which doesn't scale for
disconnected/air-gapped operator mirroring workflows.
Fix
- Identify the build step (introduced between v26.3.3 and v26.7.0) that
adds the FIPS/OpenSSL config layer, and ensure it's gzip-compressed
like every other layer. - Revert to publishing Docker v2 Schema2 manifest/manifest-list media
types (as in v25.10.0/v26.3.3/v26.7.0) instead of OCI-only, given the
broader ecosystem's Docker v2s2 compatibility expectations for
certified/catalog content.
Environment
- Images affected:
nvcr.io/nvidia/gpu-operator(and likely other
nvidia/cloud-native/*child images referenced by the certified
gpu-operator-bundle) - Versions: v25.10.0, v26.3.3, v26.7.0, v26.7.1, v26.7.2 (current latest)
- Tools used to reproduce:
skopeo1.22.3,jq,file,tar,curl
Metadata
Metadata
Assignees
Labels
bugIssue/PR to expose/discuss/fix a bugIssue/PR to expose/discuss/fix a bug
Activity
rahulait commentedon Nov 4, 2026
Thanks @nikitabugrovsky , we are looking into this.
For switching manifeststype to OCI:
We believe this happened because buildkit enabled oci-mediatypes=true by default in moby/buildkit#6824 which was in June, 2026
Later, docker engine 29.6.0 picked up this buildkit in its release: https://docs.docker.com/engine/release-notes/29/#2960
Its possible newer images are built on newer version of docker and hence manifest type has changed to OCI.
For uncompressed layer, we are investigating why there is an uncompressed layer in the image.
rahulait commentedon Nov 4, 2026
We are fixing the uncompressed layers in 26.7.3. We do not currently plan a blanket revert to Docker v2 Schema2. The reported schema error specifically rejects an uncompressed Docker layer descriptor (application/vnd.docker.image.rootfs.diff.tar), and changing manifest formats alone would not resolve that. Our planned fix addresses layer compression; we have not established that OCI media types also need to change.
rahulait commentedon Nov 6, 2026
Hi @nikitabugrovsky , we are in the process of building images with the fix for v26.7.3 release. Can you please analyze these sample images and let us know if they fix the issue you are seeing or not. These images should not have any uncompressed layer in them.
nikitabugrovsky commentedon Nov 11, 2026
Thanks @rahulait, we confirmed the fix.
All layers in the 4 sample images are now gzip-compressed (no uncompressed
tarlayers remain):We also did internal testing with
oc-mirroragainst real Quay instances (3.14.9, 3.16.2, 3.16.6), including a known-failing image (nvcr.io/nvidia/cloud-native/gdrdrv) as a control group in the sameImageSetConfiguration. Result: the 4 new images mirror successfully with no errors, while the control image still fails as expected, confirming the fix resolves the mirroring issue.nikitabugrovsky commentedon Nov 16, 2026
Sorry for the ambiguity. The floating tag
v2.6-rhel9currently points to a different, newer digest than the one we actually tested. The digest we used in our control group wassha256:eef1b6663d78555f0e5c49c156196b67e8a3008bc22aaa02b03dad53887ee0f1, which corresponds to the specific tagv2.6-rhel9.6, notv2.6-rhel9.All layers in
v2.6-rhel9.6are already gzip-compressed. The problem is a type mismatch between the top-level index and its children:So the index says "OCI" at the top, but every manifest it references is pure Docker Schema2 underneath, compressed and all. That inconsistency is what triggers the same client-side refusal as the original issue:
This is why I said the fix needs to be "unified" typing: the manifest list/index media type and every child manifest/layer it points to need to agree on Docker-vs-OCI, consistently.
v2.6-rhel9(the newer, floating tag) looks fine on this front (pure Docker list, pure Docker children), so it seems like this was already fixed going forward, it's just that the older pinnedv2.6-rhel9.6build still has the mismatch. Worth checking whether any other still-referenced pinned tags across thenvidia/cloud-native/*images have the same latent inconsistency.