Skip to content

[Bug]: gpu-operator images v26.7.0+, contain an uncompressed layer; v26.7.1+ also switched to OCI manifest types breaks Quay/oc-mirror mirroring #3166

Open

Description

@nikitabugrovsky

Evidence

1. Manifest-list media type per version

$ for tag in v25.10.0 v26.3.3 v26.7.0 v26.7.1 v26.7.2; do
    echo "=== $tag ==="
    skopeo inspect --raw docker://nvcr.io/nvidia/gpu-operator:$tag | jq -r .mediaType
  done
=== v25.10.0 ===
application/vnd.docker.distribution.manifest.list.v2+json
=== v26.3.3 ===
application/vnd.docker.distribution.manifest.list.v2+json
=== v26.7.0 ===
application/vnd.docker.distribution.manifest.list.v2+json
=== v26.7.1 ===
application/vnd.oci.image.index.v1+json
=== v26.7.2 ===
application/vnd.oci.image.index.v1+json

2. Layer media types per version (amd64)

$ for tag in v25.10.0 v26.3.3 v26.7.0 v26.7.1 v26.7.2; do
    echo "=== $tag ==="
    digest=$(skopeo inspect --raw docker://nvcr.io/nvidia/gpu-operator:$tag \
      | jq -r '.manifests[] | select(.platform.architecture=="amd64") | .digest')
    skopeo inspect --raw docker://nvcr.io/nvidia/gpu-operator@$digest \
      | jq -r '.layers[].mediaType' | sort | uniq -c
  done
=== v25.10.0 ===
     36 application/vnd.docker.image.rootfs.diff.tar.gzip
=== v26.3.3 ===
     42 application/vnd.docker.image.rootfs.diff.tar.gzip
=== v26.7.0 ===
      1 application/vnd.docker.image.rootfs.diff.tar
     42 application/vnd.docker.image.rootfs.diff.tar.gzip
=== v26.7.1 ===
      1 application/vnd.oci.image.layer.v1.tar
     42 application/vnd.oci.image.layer.v1.tar+gzip
=== v26.7.2 ===
      1 application/vnd.oci.image.layer.v1.tar
     42 application/vnd.oci.image.layer.v1.tar+gzip

3. The uncompressed layer is confirmed at the byte level, not just a label

$ TOKEN=$(curl -s "https://nvcr.io/token?service=nvcr.io&scope=repository:nvidia/gpu-operator:pull" \
    | jq -r .token)

$ amd64_digest=$(skopeo inspect --raw docker://nvcr.io/nvidia/gpu-operator:v26.7.2 \
    | jq -r '.manifests[] | select(.platform.architecture=="amd64") | .digest')

$ layer_digest=$(skopeo inspect --raw docker://nvcr.io/nvidia/gpu-operator@$amd64_digest \
    | jq -r '.layers[] | select(.mediaType | contains("gzip") | not) | .digest')

$ curl -sL -H "Authorization: Bearer $TOKEN" \
    "https://nvcr.io/v2/nvidia/gpu-operator/blobs/$layer_digest" -o layer.bin
$ file -b layer.bin
POSIX tar archive (GNU)
$ tar -tvf layer.bin
etc/
etc/ssl/
etc/ssl/fipsmodule.cnf
etc/ssl/openssl-fips.cnf
etc/ssl/openssl-nofips.cnf

This same layer (containing OpenSSL/FIPS config files) is present,
uncompressed, in v26.7.0, v26.7.1, and v26.7.2. It does not exist at all
in v25.10.0 or v26.3.3.

Impact

Customers mirroring certified operator bundles (e.g. via oc-mirror) to
a disconnected Red Hat Quay registry fail with:

    [ERROR] : [Worker] error mirroring image
    docker://nvcr.io/nvidia/cloud-native/<image>@sha256:...
    error: Manifest list must be converted to type
    "application/vnd.docker.distribution.manifest.list.v2+json" ...
    "Instructed to preserve digests"

The only workaround today is a manual skopeo copy --format v2s2 --multi-arch=all re-push per image, which doesn't scale for
disconnected/air-gapped operator mirroring workflows.

Fix

  1. Identify the build step (introduced between v26.3.3 and v26.7.0) that
    adds the FIPS/OpenSSL config layer, and ensure it's gzip-compressed
    like every other layer.
  2. Revert to publishing Docker v2 Schema2 manifest/manifest-list media
    types (as in v25.10.0/v26.3.3/v26.7.0) instead of OCI-only, given the
    broader ecosystem's Docker v2s2 compatibility expectations for
    certified/catalog content.

Environment

  • Images affected: nvcr.io/nvidia/gpu-operator (and likely other
    nvidia/cloud-native/* child images referenced by the certified
    gpu-operator-bundle)
  • Versions: v25.10.0, v26.3.3, v26.7.0, v26.7.1, v26.7.2 (current latest)
  • Tools used to reproduce: skopeo 1.22.3, jq, file, tar, curl

Activity

  1. added
    bugIssue/PR to expose/discuss/fix a bug
    needs-triageissue or PR has not been assigned a priority-px label
    on Nov 3, 2026
  2. removed
    needs-triageissue or PR has not been assigned a priority-px label
    on Nov 4, 2026
  3. rahulait commented on Nov 4, 2026

    @rahulait
    Contributor

    Thanks @nikitabugrovsky , we are looking into this.

    For switching manifeststype to OCI:
    We believe this happened because buildkit enabled oci-mediatypes=true by default in moby/buildkit#6824 which was in June, 2026
    Later, docker engine 29.6.0 picked up this buildkit in its release: https://docs.docker.com/engine/release-notes/29/#2960

    Its possible newer images are built on newer version of docker and hence manifest type has changed to OCI.

    For uncompressed layer, we are investigating why there is an uncompressed layer in the image.

  4. rahulait commented on Nov 4, 2026

    @rahulait
    Contributor

    We are fixing the uncompressed layers in 26.7.3. We do not currently plan a blanket revert to Docker v2 Schema2. The reported schema error specifically rejects an uncompressed Docker layer descriptor (application/vnd.docker.image.rootfs.diff.tar), and changing manifest formats alone would not resolve that. Our planned fix addresses layer compression; we have not established that OCI media types also need to change.

  5. rahulait commented on Nov 6, 2026

    @rahulait
    Contributor

    Hi @nikitabugrovsky , we are in the process of building images with the fix for v26.7.3 release. Can you please analyze these sample images and let us know if they fix the issue you are seeing or not. These images should not have any uncompressed layer in them.

    ghcr.io/nvidia/container-toolkit:adbd28c3
    ghcr.io/nvidia/gpu-operator:00510bf8
    ghcr.io/nvidia/k8s-device-plugin:4416ed10
    ghcr.io/nvidia/k8s-mig-manager:84e2eca3
    
  6. nikitabugrovsky commented on Nov 11, 2026

    @nikitabugrovsky
    Author

    Thanks @rahulait, we confirmed the fix.

    All layers in the 4 sample images are now gzip-compressed (no uncompressed tar layers remain):

    ghcr.io/nvidia/container-toolkit:adbd28c3
    ghcr.io/nvidia/gpu-operator:00510bf8
    ghcr.io/nvidia/k8s-device-plugin:4416ed10
    ghcr.io/nvidia/k8s-mig-manager:84e2eca3
    

    We also did internal testing with oc-mirror against real Quay instances (3.14.9, 3.16.2, 3.16.6), including a known-failing image (nvcr.io/nvidia/cloud-native/gdrdrv) as a control group in the same ImageSetConfiguration. Result: the 4 new images mirror successfully with no errors, while the control image still fails as expected, confirming the fix resolves the mirroring issue.

  7. nikitabugrovsky commented on Nov 16, 2026

    @nikitabugrovsky
    Author

    Sorry for the ambiguity. The floating tag v2.6-rhel9 currently points to a different, newer digest than the one we actually tested. The digest we used in our control group was sha256:eef1b6663d78555f0e5c49c156196b67e8a3008bc22aaa02b03dad53887ee0f1, which corresponds to the specific tag v2.6-rhel9.6, not v2.6-rhel9.

    $ skopeo inspect docker://nvcr.io/nvidia/cloud-native/gdrdrv:v2.6-rhel9 | jq -r .Digest
    sha256:a82af76a494a9262246e73fea0c30451e52b4940a21c6278a1c25e0ac4bf8cf0
    
    $ skopeo inspect docker://nvcr.io/nvidia/cloud-native/gdrdrv:v2.6-rhel9.6 | jq -r .Digest
    sha256:eef1b6663d78555f0e5c49c156196b67e8a3008bc22aaa02b03dad53887ee0f1
    

    All layers in v2.6-rhel9.6 are already gzip-compressed. The problem is a type mismatch between the top-level index and its children:

    # 1. Top-level manifest is an OCI index
    $ skopeo inspect --raw docker://nvcr.io/nvidia/cloud-native/gdrdrv:v2.6-rhel9.6 | jq -r .mediaType
    application/vnd.oci.image.index.v1+json
    
    # 2. But the per-arch manifests it points to are Docker Schema2, not OCI
    $ skopeo inspect --raw docker://nvcr.io/nvidia/cloud-native/gdrdrv:v2.6-rhel9.6 | jq '.manifests[] | {platform, mediaType, digest}'
    {
      "platform": { "architecture": "amd64", "os": "linux" },
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:165061a7ac80ecfc1d5400c60ddd4a6ed8b47968203f48550b0bc348aff792a1"
    }
    {
      "platform": { "architecture": "arm64", "os": "linux" },
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:017f923755ec5d8f8cbaa5d7f7c9f982b95b5aa9396965e7673619d3552a770c"
    }
    
    # 3. And the amd64 manifest itself (and its layers/config) is fully Docker-typed, already compressed
    $ skopeo inspect --raw docker://nvcr.io/nvidia/cloud-native/gdrdrv@sha256:165061a7ac80ecfc1d5400c60ddd4a6ed8b47968203f48550b0bc348aff792a1 \
      | jq '{mediaType, configMediaType: .config.mediaType, layerMediaTypes: [.layers[].mediaType] | unique}'
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "configMediaType": "application/vnd.docker.container.image.v1+json",
      "layerMediaTypes": ["application/vnd.docker.image.rootfs.diff.tar.gzip"]
    }
    

    So the index says "OCI" at the top, but every manifest it references is pure Docker Schema2 underneath, compressed and all. That inconsistency is what triggers the same client-side refusal as the original issue:

    $ skopeo copy --all --format v2s2 --preserve-digests \
        docker://nvcr.io/nvidia/cloud-native/gdrdrv:v2.6-rhel9.6 \
        docker://<any-registry>/test/gdrdrv:v2.6-rhel9.6 --dest-tls-verify=false
    ...
    level=fatal msg="Manifest list must be converted to type \"application/vnd.docker.distribution.manifest.list.v2+json\" to be written to destination, but we cannot modify it: \"Instructed to preserve digests\""
    

    This is why I said the fix needs to be "unified" typing: the manifest list/index media type and every child manifest/layer it points to need to agree on Docker-vs-OCI, consistently. v2.6-rhel9 (the newer, floating tag) looks fine on this front (pure Docker list, pure Docker children), so it seems like this was already fixed going forward, it's just that the older pinned v2.6-rhel9.6 build still has the mismatch. Worth checking whether any other still-referenced pinned tags across the nvidia/cloud-native/* images have the same latent inconsistency.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugIssue/PR to expose/discuss/fix a bug

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions