Skip to content

fix(cdi): Include 32-bit libraries in discovery - #2035

Open
ehfd wants to merge 3 commits into
NVIDIA:mainfrom
ehfd:ehfd
Open

ehfd wants to merge 3 commits into
NVIDIA:mainfrom
ehfd:ehfd

Conversation

@ehfd

@ehfd ehfd commented Aug 29, 2026 •

Copy link
Copy Markdown
Contributor

Superseding and takeover of #1968.

Closes #1968

Reviewer: @henry118 @cdesiniotis

Continues #1968 by @elibosley, reworked after the review: his discovery change is replaced by a separate 32-bit locator, and he is credited as co-author of that commit. Related to #563.

Background

ldcache.List() returns the 32-bit and 64-bit library sets separately, and the CDI path discards the 32-bit one, so a generated spec never carries the driver's 32-bit libraries. A 32-bit application (a Steam or Proton title, a 32-bit CUDA or Vulkan consumer) then falls back to software rendering or fails to start, where the legacy runtime mounts them for a container that requests the compat32 driver capability.

What this does

  • A compat32 mode. NewLibrary32Locator locates through the ldcache's 32-bit entries, and a new gated mode, like gds and mofed, uses it to discover the driver's 32-bit libraries (those with its version suffix). NewLibraryLocator and the 64-bit discovery are unchanged.
  • A directory of their own. The libraries are mounted into /usr/lib/nvidia/compat32, not at their host paths, so a RHEL host's /usr/lib cannot land on an Arch container's 64-bit /usr/lib. The mode's update-ldcache hook adds the directory to the ldcache, where glibc's dynamic linker uses it for 32-bit programs only. The hook never writes it to musl's .path file, which carries no architecture.
  • Only for containers that ask. In jit-cdi mode a container gets the mode's device when it requests the compat32 driver capability, by name or through all: the request that makes the legacy runtime pass --compat32 to nvidia-container-cli. Every other container gets what it gets today. Nothing in the container is inspected; the request is the gate.

With static specs, the device is generated on request and asked for beside the GPUs:

$ nvidia-ctk cdi generate --mode=compat32 --class=compat32 --output=/etc/cdi/nvidia-compat32.yaml
$ docker run --device nvidia.com/gpu=all --device nvidia.com/compat32=all ...

musl

The hook also writes musl's .path file differently, which fixes a pre-existing bug. It used to append the driver directories: Alpine ships no .path file, and creating one replaces musl's built-in search path, so /usr/local/lib was dropped; every update appended the system directories again; and directories already in ld.so.conf were left out, though musl does not read it. The file is now rewritten with the driver directories that are not searched already, ahead of the existing entries as the 00-nvcr-*.conf drop-in puts them for glibc; then the existing entries, split on colons and newlines as musl does, with the built-in search path standing in for a missing file; then the system directories, each once. musl is also detected by its dynamic linker, not only by /etc/alpine-release.

The review

  1. Opt-in: nothing is mounted unless the container requests compat32 (jit-cdi) or the compat32 device (static specs). The default spec is unchanged.
  2. A separate locator: NewLibrary32Locator and the compat32 discoverer; NewLibraryLocator keeps First, and the hook no longer reads ELF headers.
  3. A dedicated directory: /usr/lib/nvidia/compat32, in the ldcache of the containers that have it.
  4. The container-side gate: only the explicit request. The .path file of a musl container never gets the directory.

Verification

Bare-metal x86_64, Ubuntu 26.04, 2x Tesla P100-SXM2-16GB, driver 580.178.04, libnvidia-gl-580:i386 installed, Docker 29.7.2, runc. The probe image is ubuntu:26.04 with libc6:i386, libvulkan1{,:i386}, libegl1{,:i386}, libx11-6{,:i386} and libxext6{,:i386}; a probe built for both word sizes opens libcuda.so.1 (cuInit, cuDeviceGetCount) and libvulkan.so.1 (vkCreateInstance, vkEnumeratePhysicalDevices).

jit-cdi: this branch's nvidia-container-runtime (mode jit-cdi, its own config) creating an OCI bundle of a fresh copy of the image with runc, NVIDIA_VISIBLE_DEVICES=all:

NVIDIA_DRIVER_CAPABILITIES 32-bit CUDA 32-bit Vulkan /usr/lib/nvidia/compat32
compute,utility libcuda.so.1 not found VK_ERROR_INCOMPATIBLE_DRIVER absent
graphics,compat32 both P100s both P100s 20 libraries
all both P100s both P100s 20 libraries
all, with the installed 1.20.1 libcuda.so.1 not found VK_ERROR_INCOMPATIBLE_DRIVER absent

The 64-bit probe sees both P100s through CUDA and Vulkan in every row.

Static specs: nvidia-ctk cdi generate --mode=compat32 --class=compat32 gives one compat32=all device with the 20 libraries from /usr/lib/i386-linux-gnu mounted into /usr/lib/nvidia/compat32 and one update-ldcache hook for that directory. docker run --device nvidia.com/gpu=all alone leaves the 32-bit probe failing as before; with --device nvidia.com/compat32=all it sees both P100s through CUDA and Vulkan, and ldconfig -p lists libcuda.so.1 (libc6) => /usr/lib/nvidia/compat32/libcuda.so.1 beside the x86-64 entry.

musl: in alpine:3.22 with both devices and this branch's hook, /etc/ld-musl-x86_64.path is /usr/lib/x86_64-linux-gnu, /usr/lib/x86_64-linux-gnu/vdpau, /lib, /usr/local/lib, /usr/lib, /lib64, /usr/lib64, /libx32, /usr/libx32, with no compat32 entry. With 1.20.1's hook it lacks /usr/local/lib and lists the system directories twice, once for each device's hook.

Unit tests cover the 32-bit ldcache selection, the discoverer's mounts and hook, the .path file (new and existing, ordering, musl detection, the compat32 directory left out), and the jit-cdi request. go test ./... passes (cmd/nvidia-container-runtime with its binary on PATH), and golangci-lint (0 issues), gofmt -s and goimports -local are clean.

Notes for reviewers

  • 32-bit VDPAU (libvdpau_nvidia in /usr/lib/i386-linux-gnu/vdpau) is not covered, since the ldcache does not list it.
  • The 32-bit locator reads the ldcache only, so hosts without one (NixOS drives nvidia-ctk with --library-search-path) get no 32-bit libraries from this mode.
  • nvidia-ctk cdi generate applies --class (default gpu) to every mode, as it does for gds, hence --class=compat32 above.

Copilot AI lite review requested due to automatic review settings August 29, 2026 04:40
@copy-pr-bot

copy-pr-bot Bot commented Aug 29, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@elibosley

Copy link
Copy Markdown

Thank you! I apologize for missing the earlier comments on my PR.

@ehfd

ehfd commented Aug 29, 2026

Copy link
Copy Markdown
Contributor Author

@elibosley You're the author in the commits, so it's just a carry-over. I'll handle the rest! Thank you for your contribution!

@myeolenv

Copy link
Copy Markdown

Thanks for the contribution. We will review this PR soon.

@ehfd

ehfd commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor Author

@henry118 Just a cordial reminder. I just would like this before v1.20.1 to blend with the other PRs.

@ehfd
ehfd force-pushed the ehfd branch 2 times, most recently from 0f8157f to 39c162f Compare September 5, 2026 08:49
@ehfd

ehfd commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

@henry118 I reduced the code footprint dramatically while doing the same work. FYI.

@henry118 henry118 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ehfd Thanks for bearing with me. A few thoughts on the general approach:

  1. I still think we want a flag here to gate this feature. Unconditionally discovering and mounting the full 32-bit lib set for every container is probably unnecessary for most use cases;
  2. IMO we can skip the 32-bit injection for alpine entirely. This feels like solving for a use case we would never support (32-bit containers), and the host is always assumed 64-bit.

@ehfd

ehfd commented Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

I found that i386 multiarch is not a supported concept in musl; understood.

@ehfd

ehfd commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor Author

@henry118

I made a compromise where, host-wise, 32-bit libraries are on by default in CDI specs when available, explicitly disabled by nvidia-ctk cdi generate --disable-compat32, because if the guard is host-wide, nobody can use it in K8s unless the administrators set it.

Otherwise, the condition to provision inside the container is when:

  • the container does not use musl, and
  • it requested the compat32/all driver capability through NVIDIA_DRIVER_CAPABILITIES; the capability that drives --compat32 in nvidia-container-cli, or it ships a dynamic linker for 32-bit applications (/lib/ld-linux.so.2 or /lib32/ld-linux.so.2; /lib/ld-linux-armhf.so.3 or /lib/ld-linux.so.3 on arm64).

@ehfd

ehfd commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

Associated to #2049 ?

@ehfd

ehfd commented Sep 22, 2026

Copy link
Copy Markdown
Contributor Author

@cdesiniotis @henry118 Conflicts resolved and rebased.

@ehfd

ehfd commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

@henry118 @cdesiniotis Hi, any status on this PR? Just a cordial reminder.

@henry118

Copy link
Copy Markdown
Member

@henry118 @cdesiniotis Hi, any status on this PR? Just a cordial reminder.

@ehfd thanks for the reminder. I'll get to this later this week.

@henry118 henry118 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ehfd Thanks for bearing with me and putting this together.

I think we need to talk through the approach a bit more before going line by line though. A few thoughts:

  1. Can we make this opt-in? Right now the 32-bit libs get mounted into every container by default, and the hook only decides whether they show up in the ldcache. I'd rather keep today's behavior as the default.

  2. Changing NewLibraryLocator from First to Merge touches a lot more than the 32-bit case. What about a separate locator just for the 32-bit libs, with its own discoverer? That keeps the 64-bit path untouched, and the hook wouldn't have to sniff ELF headers to figure out which dirs are 32-bit.

  3. Mounting at host paths can bite us. On a RHEL host the 32-bit libs are in /usr/lib, so in an Arch container (/usr/lib64 -> /usr/lib) the 32-bit and 64-bit libcuda end up on the same path. nvidia-container-cli gets around this by looking at the container's layout, which we can't do with a static spec. Could we mount them into a dedicated dir instead (something like /usr/lib/nvidia/compat32) and only add it to the ldcache when it's needed?

  4. If it's opt-in and the libs live in their own dir, I'm not sure we need the container-side gate at all. glibc already ignores wrong-class entries in the ldcache, and musl just wouldn't get the dir added to its .path file.

@ehfd

ehfd commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor Author

@henry118

Can we make this opt-in? Right now the 32-bit libs get mounted into every container by default, and the hook only decides whether they show up in the ldcache

Yeah, this is actually a bug that should be fixed, but I still think the conditions for being mounted inside the container that the user can trigger should still be applied (otherwise, if this feature needs to be explicitly configured by the cluster administrators, a vast majority of users will be locked out of this feature when not configured):

Otherwise, the condition to provision inside the container is when:

  • the container does not use musl, and
  • it requested the compat32/all driver capability through NVIDIA_DRIVER_CAPABILITIES; the capability that drives --compat32 in nvidia-container-cli, or it ships a dynamic linker for 32-bit applications (/lib/ld-linux.so.2 or /lib32/ld-linux.so.2; /lib/ld-linux-armhf.so.3 or /lib/ld-linux.so.3 on arm64).

Changing NewLibraryLocator from First to Merge touches a lot more than the 32-bit case. What about a separate locator just for the 32-bit libs, with its own discoverer? That keeps the 64-bit path untouched, and the hook wouldn't have to sniff ELF headers to figure out which dirs are 32-bit.

Got it.


Mounting at host paths can bite us. On a RHEL host the 32-bit libs are in /usr/lib, so in an Arch container (/usr/lib64 -> /usr/lib) the 32-bit and 64-bit libcuda end up on the same path. nvidia-container-cli gets around this by looking at the container's layout, which we can't do with a static spec. Could we mount them into a dedicated dir instead (something like /usr/lib/nvidia/compat32) and only add it to the ldcache when it's needed?

Got it.


If it's opt-in and the libs live in their own dir, I'm not sure we need the container-side gate at all. glibc already ignores wrong-class entries in the ldcache, and musl just wouldn't get the dir added to its .path file.

Have to think of a resolution in this case, not sure right now.

@ehfd

ehfd commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

@henry118 Reworked along your review, rebased on main:

  1. Nothing is mounted unless the container asks: in jit-cdi mode, the compat32 driver capability (by name or through all, as the legacy runtime's --compat32); with static specs, a nvidia.com/compat32=all device generated with --mode=compat32. The default spec is unchanged.
  2. A separate NewLibrary32Locator over the ldcache's 32-bit entries and a compat32 discoverer, as a gated mode beside gds and mofed. NewLibraryLocator keeps First, and the hook no longer reads ELF headers.
  3. The libraries are mounted into /usr/lib/nvidia/compat32, which the mode's update-ldcache hook adds to the ldcache.
  4. The explicit request is the only gate; nothing in the rootfs is inspected. musl's .path file never gets the directory.

The description has the runs on a P100 host: jit-cdi with compute,utility, graphics,compat32, all and 1.20.1 for comparison, the static compat32 device, and an Alpine container.

@ehfd
ehfd requested a review from henry118 October 20, 2026 09:52
@ehfd

ehfd commented Oct 27, 2026

Copy link
Copy Markdown
Contributor Author

@henry118 Just a reminder. Thank you.

ehfd and others added 3 commits November 14, 2026 19:44
ldcache.List() returns the 32-bit and 64-bit library sets separately, and the
library locator keeps the 64-bit one. NewLibrary32Locator locates through the
32-bit set alone, so a discoverer can find the 32-bit driver libraries without
changing how the native ones are found.

Co-authored-by: Eli Bosley <eli@bosley.dev>
Signed-off-by: Seungmin Kim <8457324+ehfd@users.noreply.github.com>
…st compat32

A compat32 mode discovers the driver's 32-bit libraries and mounts them into
/usr/lib/nvidia/compat32, which the update-ldcache hook adds to the container's
ldcache; they are not mounted at their host paths, which can be 64-bit
directories in the container. musl's .path file never gets the directory.

In jit-cdi mode a container gets them when it requests the compat32 driver
capability, by name or through "all", as the legacy runtime passes --compat32
to nvidia-container-cli then. A static spec is generated with
`nvidia-ctk cdi generate --mode=compat32 --class=compat32` and requested as
nvidia.com/compat32=all beside the GPUs.

Signed-off-by: Seungmin Kim <8457324+ehfd@users.noreply.github.com>
The hook appended the driver directories to musl's .path file. Alpine ships
none, and creating one replaces musl's built-in search path, so /usr/local/lib
was dropped. Each update also appended the system directories again, and the
directories already listed in ld.so.conf were left out, though musl does not
read it.

Rewrite the file instead: the driver directories that are not searched already,
ahead of the existing entries as the 00-nvcr-*.conf drop-in puts them for glibc,
then the existing entries (split on colons and newlines, as musl does, with its
built-in search path standing in for a missing file), then the system
directories, each once. musl is also detected by its dynamic linker, not only by
/etc/alpine-release.

Signed-off-by: Seungmin Kim <8457324+ehfd@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants