Repository navigation
Conversation
|
Thank you! I apologize for missing the earlier comments on my PR. |
|
@elibosley You're the author in the commits, so it's just a carry-over. I'll handle the rest! Thank you for your contribution! |
|
Thanks for the contribution. We will review this PR soon. |
|
@henry118 Just a cordial reminder. I just would like this before v1.20.1 to blend with the other PRs. |
0f8157f to
39c162f
Compare
|
@henry118 I reduced the code footprint dramatically while doing the same work. FYI. |
henry118
left a comment
There was a problem hiding this comment.
@ehfd Thanks for bearing with me. A few thoughts on the general approach:
- I still think we want a flag here to gate this feature. Unconditionally discovering and mounting the full 32-bit lib set for every container is probably unnecessary for most use cases;
- IMO we can skip the 32-bit injection for alpine entirely. This feels like solving for a use case we would never support (32-bit containers), and the host is always assumed 64-bit.
|
I found that i386 multiarch is not a supported concept in musl; understood. |
|
I made a compromise where, host-wise, 32-bit libraries are on by default in CDI specs when available, explicitly disabled by Otherwise, the condition to provision inside the container is when:
|
|
Associated to #2049 ? |
|
@cdesiniotis @henry118 Conflicts resolved and rebased. |
|
@henry118 @cdesiniotis Hi, any status on this PR? Just a cordial reminder. |
@ehfd thanks for the reminder. I'll get to this later this week. |
henry118
left a comment
There was a problem hiding this comment.
@ehfd Thanks for bearing with me and putting this together.
I think we need to talk through the approach a bit more before going line by line though. A few thoughts:
-
Can we make this opt-in? Right now the 32-bit libs get mounted into every container by default, and the hook only decides whether they show up in the ldcache. I'd rather keep today's behavior as the default.
-
Changing
NewLibraryLocatorfromFirsttoMergetouches a lot more than the 32-bit case. What about a separate locator just for the 32-bit libs, with its own discoverer? That keeps the 64-bit path untouched, and the hook wouldn't have to sniff ELF headers to figure out which dirs are 32-bit. -
Mounting at host paths can bite us. On a RHEL host the 32-bit libs are in
/usr/lib, so in an Arch container (/usr/lib64 -> /usr/lib) the 32-bit and 64-bitlibcudaend up on the same path. nvidia-container-cli gets around this by looking at the container's layout, which we can't do with a static spec. Could we mount them into a dedicated dir instead (something like/usr/lib/nvidia/compat32) and only add it to the ldcache when it's needed? -
If it's opt-in and the libs live in their own dir, I'm not sure we need the container-side gate at all. glibc already ignores wrong-class entries in the ldcache, and musl just wouldn't get the dir added to its .path file.
Yeah, this is actually a bug that should be fixed, but I still think the conditions for being mounted inside the container that the user can trigger should still be applied (otherwise, if this feature needs to be explicitly configured by the cluster administrators, a vast majority of users will be locked out of this feature when not configured):
Got it.
Got it.
Have to think of a resolution in this case, not sure right now. |
|
@henry118 Reworked along your review, rebased on
The description has the runs on a P100 host: |
|
@henry118 Just a reminder. Thank you. |
ldcache.List() returns the 32-bit and 64-bit library sets separately, and the library locator keeps the 64-bit one. NewLibrary32Locator locates through the 32-bit set alone, so a discoverer can find the 32-bit driver libraries without changing how the native ones are found. Co-authored-by: Eli Bosley <eli@bosley.dev> Signed-off-by: Seungmin Kim <8457324+ehfd@users.noreply.github.com>
…st compat32 A compat32 mode discovers the driver's 32-bit libraries and mounts them into /usr/lib/nvidia/compat32, which the update-ldcache hook adds to the container's ldcache; they are not mounted at their host paths, which can be 64-bit directories in the container. musl's .path file never gets the directory. In jit-cdi mode a container gets them when it requests the compat32 driver capability, by name or through "all", as the legacy runtime passes --compat32 to nvidia-container-cli then. A static spec is generated with `nvidia-ctk cdi generate --mode=compat32 --class=compat32` and requested as nvidia.com/compat32=all beside the GPUs. Signed-off-by: Seungmin Kim <8457324+ehfd@users.noreply.github.com>
The hook appended the driver directories to musl's .path file. Alpine ships none, and creating one replaces musl's built-in search path, so /usr/local/lib was dropped. Each update also appended the system directories again, and the directories already listed in ld.so.conf were left out, though musl does not read it. Rewrite the file instead: the driver directories that are not searched already, ahead of the existing entries as the 00-nvcr-*.conf drop-in puts them for glibc, then the existing entries (split on colons and newlines, as musl does, with its built-in search path standing in for a missing file), then the system directories, each once. musl is also detected by its dynamic linker, not only by /etc/alpine-release. Signed-off-by: Seungmin Kim <8457324+ehfd@users.noreply.github.com>
Superseding and takeover of #1968.
Closes #1968
Reviewer: @henry118 @cdesiniotis
Continues #1968 by @elibosley, reworked after the review: his discovery change is replaced by a separate 32-bit locator, and he is credited as co-author of that commit. Related to #563.
Background
ldcache.List()returns the 32-bit and 64-bit library sets separately, and the CDI path discards the 32-bit one, so a generated spec never carries the driver's 32-bit libraries. A 32-bit application (a Steam or Proton title, a 32-bit CUDA or Vulkan consumer) then falls back to software rendering or fails to start, where the legacy runtime mounts them for a container that requests thecompat32driver capability.What this does
compat32mode.NewLibrary32Locatorlocates through the ldcache's 32-bit entries, and a new gated mode, likegdsandmofed, uses it to discover the driver's 32-bit libraries (those with its version suffix).NewLibraryLocatorand the 64-bit discovery are unchanged./usr/lib/nvidia/compat32, not at their host paths, so a RHEL host's/usr/libcannot land on an Arch container's 64-bit/usr/lib. The mode'supdate-ldcachehook adds the directory to the ldcache, where glibc's dynamic linker uses it for 32-bit programs only. The hook never writes it to musl's.pathfile, which carries no architecture.jit-cdimode a container gets the mode's device when it requests thecompat32driver capability, by name or throughall: the request that makes the legacy runtime pass--compat32tonvidia-container-cli. Every other container gets what it gets today. Nothing in the container is inspected; the request is the gate.With static specs, the device is generated on request and asked for beside the GPUs:
musl
The hook also writes musl's
.pathfile differently, which fixes a pre-existing bug. It used to append the driver directories: Alpine ships no.pathfile, and creating one replaces musl's built-in search path, so/usr/local/libwas dropped; every update appended the system directories again; and directories already inld.so.confwere left out, though musl does not read it. The file is now rewritten with the driver directories that are not searched already, ahead of the existing entries as the00-nvcr-*.confdrop-in puts them for glibc; then the existing entries, split on colons and newlines as musl does, with the built-in search path standing in for a missing file; then the system directories, each once. musl is also detected by its dynamic linker, not only by/etc/alpine-release.The review
compat32(jit-cdi) or thecompat32device (static specs). The default spec is unchanged.NewLibrary32Locatorand thecompat32discoverer;NewLibraryLocatorkeepsFirst, and the hook no longer reads ELF headers./usr/lib/nvidia/compat32, in the ldcache of the containers that have it..pathfile of a musl container never gets the directory.Verification
Bare-metal x86_64, Ubuntu 26.04, 2x Tesla P100-SXM2-16GB, driver 580.178.04,
libnvidia-gl-580:i386installed, Docker 29.7.2, runc. The probe image isubuntu:26.04withlibc6:i386,libvulkan1{,:i386},libegl1{,:i386},libx11-6{,:i386}andlibxext6{,:i386}; a probe built for both word sizes openslibcuda.so.1(cuInit,cuDeviceGetCount) andlibvulkan.so.1(vkCreateInstance,vkEnumeratePhysicalDevices).jit-cdi: this branch'snvidia-container-runtime(modejit-cdi, its own config) creating an OCI bundle of a fresh copy of the image with runc,NVIDIA_VISIBLE_DEVICES=all:NVIDIA_DRIVER_CAPABILITIES/usr/lib/nvidia/compat32compute,utilitylibcuda.so.1not foundVK_ERROR_INCOMPATIBLE_DRIVERgraphics,compat32allall, with the installed 1.20.1libcuda.so.1not foundVK_ERROR_INCOMPATIBLE_DRIVERThe 64-bit probe sees both P100s through CUDA and Vulkan in every row.
Static specs:
nvidia-ctk cdi generate --mode=compat32 --class=compat32gives onecompat32=alldevice with the 20 libraries from/usr/lib/i386-linux-gnumounted into/usr/lib/nvidia/compat32and oneupdate-ldcachehook for that directory.docker run --device nvidia.com/gpu=allalone leaves the 32-bit probe failing as before; with--device nvidia.com/compat32=allit sees both P100s through CUDA and Vulkan, andldconfig -plistslibcuda.so.1 (libc6) => /usr/lib/nvidia/compat32/libcuda.so.1beside the x86-64 entry.musl: in
alpine:3.22with both devices and this branch's hook,/etc/ld-musl-x86_64.pathis/usr/lib/x86_64-linux-gnu,/usr/lib/x86_64-linux-gnu/vdpau,/lib,/usr/local/lib,/usr/lib,/lib64,/usr/lib64,/libx32,/usr/libx32, with nocompat32entry. With 1.20.1's hook it lacks/usr/local/liband lists the system directories twice, once for each device's hook.Unit tests cover the 32-bit ldcache selection, the discoverer's mounts and hook, the
.pathfile (new and existing, ordering, musl detection, the compat32 directory left out), and thejit-cdirequest.go test ./...passes (cmd/nvidia-container-runtimewith its binary onPATH), andgolangci-lint(0 issues),gofmt -sandgoimports -localare clean.Notes for reviewers
libvdpau_nvidiain/usr/lib/i386-linux-gnu/vdpau) is not covered, since the ldcache does not list it.nvidia-ctkwith--library-search-path) get no 32-bit libraries from this mode.nvidia-ctk cdi generateapplies--class(defaultgpu) to every mode, as it does forgds, hence--class=compat32above.