Skip to content

modifier: fix missing update-ldcache hook in legacy mode - #2150

Open
rootkiller6788 wants to merge 2 commits into
NVIDIA:mainfrom
rootkiller6788:fix-legacy-mode-ldcache-hook
Open

rootkiller6788 wants to merge 2 commits into
NVIDIA:mainfrom
rootkiller6788:fix-legacy-mode-ldcache-hook

Conversation

@rootkiller6788

Copy link
Copy Markdown

Legacy mode lost its update-ldcache hook.

getCudaCompatModeModifier() builds that hook through the ldconfig discoverer
with an empty mount discoverer, and it does so on purpose: on a legacy runtime
the CUDA compat libraries only end up in the ldconfig search path after the
enable-cuda-compat hook has run inside the container, so at spec-generation
time there is nothing for the discoverer to find.

cd69ba4 then added a guard so the discoverer returns no hook at all when it
finds no library folders. That guard is right for the CDI and CSV callers, but
it silently killed the legacy one: with cuda-compat-mode = "hook" you'd get
enable-cuda-compat and no update-ldcache, so the compat libraries were written
to /etc/ld.so.conf.d/00-compat-.conf and ldconfig never ran.

I fixed it at the one call site instead of touching the discoverer, since the
other four callers do have real libraries to discover.

Testing: I faked the driver version so the new test in gated_test.go needs no
gpu or driver on the host. It fails on main (only enable-cuda-compat comes
through) and passes with the fix. The two failures remaining in
./internal/modifier/... and ./internal/discover/... are pre-existing and
unrelated: TestNewCSVModifier/visible_devices_all and TestDrmDevicesByPath.

@copy-pr-bot

copy-pr-bot Bot commented Oct 18, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@rootkiller6788
rootkiller6788 marked this pull request as ready for review October 19, 2026 15:15

@blacksoxx blacksoxx left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tracking this down, the explanation of why the legacy call site passes discover.None{} made it easy to follow.

What I checked on b8a59b7:

  • The fix gives the same hook as before cd69ba4: hookCreator.Create(UpdateLDCacheHook) with no folders is what ldconfig.Hooks() returned for an empty mount list before the guard. With no --folder arguments, update-ldcache still runs ldconfig -f /etc/ld.so.conf, which reads the ld.so.conf.d drop-in that enable-cuda-compat writes, so the compat libraries do end up in the cache. The discover.Merge order keeps enable-cuda-compat first.
  • The guard is in v1.20.0 and v1.20.1, so this is a regression for legacy mode with cuda-compat-mode = "hook" since v1.20.0.
  • TestGetCudaCompatModeModifier fails with main's gated.go and passes with yours.

One note on the two failures you mentioned: the branch is 74 commits behind main. I merged it into current main (44f1c6e, clean merge) and go test ./internal/modifier/... ./internal/discover/... passes there, including TestNewCSVModifier and TestDrmDevicesByPath. On macOS I could only reproduce the visible_devices_all failure from a checkout under /tmp, where the expected paths come back with a /private prefix, so it may be environment-related rather than pre-existing on main. A rebase would also let CI run against the current code. golangci-lint run ./internal/modifier/... reports 0 issues on the merge.

Since cd69ba4 the ldconfig discoverer returns no hook when it finds no
library folders. The legacy-mode call site in getCudaCompatModeModifier
passes discover.None{} deliberately: the CUDA compat libraries are only
added to the ldconfig search path by the enable-cuda-compat hook while
the container starts, so there is never anything to find here.

As a result a legacy runtime with cuda-compat-mode = "hook" got the
enable-cuda-compat hook but not the update-ldcache hook, so the compat
libraries were written to /etc/ld.so.conf.d/00-compat-<uuid>.conf and
ldconfig was never run.

Create the hook with the hook creator directly instead of going through
the ldconfig discoverer.

Signed-off-by: rootkiller6788 <c8688rickowens@outlook.com>
The legacy mode cuda-compat-mode = "hook" case is the one that regressed
in cd69ba4, so make sure the update-ldcache hook is injected alongside
enable-cuda-compat. Also cover the cuda-compat-mode = "ldconfig" no-op
case and the non-legacy modes, which rely on the discovered update-ldcache
hook and must not get a second one.

The driver version is faked so the test does not need a driver on the host.

Signed-off-by: rootkiller6788 <c8688rickowens@outlook.com>
@rootkiller6788
rootkiller6788 force-pushed the fix-legacy-mode-ldcache-hook branch from b8a59b7 to 59bc8b0 Compare November 8, 2026 09:34
@rootkiller6788

Copy link
Copy Markdown
Author

Thanks for the thorough review, and for confirming the fix restores the pre-cd69ba45 behavior.

I've rebased the branch onto current main (1c9a334). The commits are now 8f0a3a7
and 59bc8b0, and the diff is unchanged: internal/modifier/gated.go (+4/-9) plus
the new internal/modifier/gated_test.go.

Re-ran everything on a Linux checkout (Ubuntu 24.04, Go 1.27.1):

  • go build ./..., go vet ./internal/modifier/, and gofmt -l on the two
    touched files are clean.
  • TestGetCudaCompatModeModifier passes (4/4).
  • TestDrmDevicesByPath passes once the testdata symlinks are actual symlinks —
    my first checkout had them materialized as plain files because of
    core.symlinks=false, which is a local Windows checkout artifact.

You were right about TestNewCSVModifier: on a clean checkout of origin/main at
1c9a334 it fails exactly the same way here. The actual spec contains an extra
nvidia-cdi-hook enable-cuda-compat --host-cuda-version=12.8 hook because this
host has CUDA 12.8 compat libraries installed, so it is environment-related
rather than something this branch introduces.

Thanks again — happy to adjust anything else you'd like.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants