You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(cuda.core): share ownership of repeat imports of an IPC memory pool - #3511
Milestone: cuda.core 1.4.0 (release note in cuda_core/docs/source/release/1.4.0-notes.rst).
Importing the same memory pool more than once in one process (for example PinnedMemoryResource.from_allocation_handle(int(fd)) twice, or two threads importing the same UUID) gave independent owners of one driver pool. Closing one of them broke the others for pinned pools, and concurrent imports of one UUID raised "internal cuda.core error ... register() returned a different memory resource".
Root cause
Every import of a pool in a process returns the same CUmemoryPool, and the driver counts the imports. But create_mempool_handle_ipc wrapped each import in its own resource handle, so one pool had two reference counts. Closing one handle ran the pool deleter's access revocation (the nvbug 5698116 workaround), and because access settings belong to the pool, that removed the device access the other import relied on. Verified with raw cuda.bindings calls on driver 595.99.02.
Fix
create_mempool_handle_ipc finds an existing owner through a registry keyed by the imported CUmemoryPool (Level 1 in REGISTRY_DESIGN.md). A repeat import adds one to the owner's import count and returns its handle. When the last reference is released, the owner revokes access once and then calls cuMemPoolDestroy once per import: destroying a duplicate import early fails with CUDA_ERROR_INVALID_VALUE while buffers imported from the pool are outstanding.
An import that finds the previous owner's release in flight waits for it. The last reference can be dropped by an imported buffer or by garbage collection, so the application cannot synchronize with it. Without the wait, the old owner's revocation removes the access a new owner granted, and its destroys fail once the new owner has imported a buffer. ReleasingHandleRegistry, a HandleRegistry variant whose entry outlives its handle until the owner's deleter calls release(), provides the wait. The deleter revokes access and destroys every import before it releases the entry.
The IPC pointer cache in deviceptr_import_ipc (the nvbug 5570902 workaround) uses the same registry instead of its own mutex and condition variable.
MP_from_allocation_handle returns the registered resource when a concurrent import of the same UUID registered first, instead of raising. MP_register holds a lock across its registry lookup and insertion, because on free-threaded builds two threads could both miss and register different resources for one UUID.
DeviceMemoryResource.from_allocation_handle sets the device in the import hook, before registration, so a concurrent import never writes to a registered resource. A registry hit keeps the resource's original device_id instead of changing it to the requested device.
Tests
New cuda_core/tests/memory_ipc/test_ipc_pool_duplicate_import.py, for device and pinned pools:
Closing one of two imports keeps the other usable.
A buffer imported through one import survives closing both, with no CUDAWarning.
Concurrent imports of one UUID return one resource, with its device set, and one pool.
A re-import that overlaps the release of the last import keeps device access and emits no CUDAWarning. A test hook, _set_ipc_pool_release_delay_for_testing, delays the old owner's cleanup so the overlap is deterministic. Without the wait, the device pool case reports failed cuMemPoolDestroy calls and the pinned case loses device access.
Against unfixed main, the pinned close test and the concurrent test also fail.
Notes for reviewers
The deleter releases the registry entry only after its destroys, not after the revocation: a woken importer could otherwise import a buffer before the old owner's destroys run, and they would fail.
The refused cuMemPoolDestroy matches a TODO in the driver's cuapiMemPoolDestroy ("Relax this and allow destruction to succeed even if memobj exists"). A pending cuMemFreeAsync of an imported pointer does not block it.
Peer access with several GPUs takes the same path but was tested on a single GPU only.
Only cuda_core is affected; neither bindings root has a counterpart.
It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.
No actionable comments were generated in the recent review.
Summary
Summary by CodeRabbit
Bug Fixes
Repeated imports of the same memory pool now share ownership. Closing one import preserves access through the others; access is revoked when the final import is released.
Outstanding buffers keep their memory pool available after its imports are closed.
Concurrent imports of the same allocation handle now return the same registered resource.
Imports that overlap with a pool’s final release wait for cleanup to complete before proceeding.
When an allocation handle is already registered, the returned resource retains its original device.
Repeated IPC memory-pool imports now share ownership until final release. The pointer cache waits for cleanup before re-importing a handle. Concurrent UUID imports return the registered resource after a type check. Tests cover duplicate imports, outstanding buffers, release races, and concurrent UUID imports.
Changes: IPC memory-pool imports
Layer / File(s)
Summary
Shared pool ownership and release cuda_core/cuda/core/_cpp/rt/api.hpp, cuda_core/cuda/core/_cpp/rt/memory.cpp, cuda_core/cuda/core/_memory/_memory_pool.pxd, cuda_core/cuda/core/_memory/_memory_pool.pyx, cuda_core/cuda/core/_memory/_pinned_memory_resource.pyx, cuda_core/cuda/core/_rt.pyx, cuda_core/cuda/core/_rt.pyi, cuda_core/tests/memory_ipc/test_ipc_pool_duplicate_import.py, cuda_core/docs/source/release/1.4.0-notes.rst
The C++ import path shares ownership for repeated pool handles and counts driver imports. Final release revokes access and destroys each import. The release-delay test hook and imported-pool access retry support the release path. Tests cover duplicate imports, outstanding buffers, and release races; release notes describe the behavior.
The Python registry serializes UUID lookup and insertion, checks the registered resource type, and returns the existing resource if another import registers first. Initialization occurs before registration. An existing device resource retains its original device. Tests check concurrent imports and device IDs.
The pointer cache uses a releasing-handle registry. An import waits while prior mapping cleanup is in progress. The design documentation describes when the registry entry is released.
Assessment against linked issues
Objective
Addressed
Explanation
Share ownership of repeated imports of the same memory pool for pinned and device resources, including while imported buffers remain open [#3510]
✅
Return the already-registered resource when concurrent imports of the same UUID race, after checking its type [#3510]
✅
Possibly related issues
NVIDIA/cuda-python#3514: The issue reports a remaining imported-pinned-pool release failure with imported pointers outstanding and states that it reproduces with this change.
Change: Bug fix · Severity of issue fixed: Medium
Merge Risk: Low · previous review 8fe6d
Repeated imports of an IPC memory pool now share ownership, which fixes premature access revocation. Remaining concerns are narrow: a deadlock under an unusual warning-handler re-import, a possible abort on an allocation failure, and a timing-based test that could pass without exercising the race. These are acceptable to follow up on, so merge risk is low.
Architecture risk: Low · up to 8fe6d
The change affects 1 system.
Changed systems:cuda_core
Architecture concerns
No architecture-level concerns identified.
Review details
Systems and components
observed — cuda_core (service) was modified; 18 changed files map to changed impact.
Before / after behavior
observed — Modified behavior in cuda_core/cuda/core/_cpp/rt/DESIGN.md: The GIL-locking rules remain, but the deviceptr_import_ipc example now describes reporting a discarded import after releasing the lock. The after_cleanup hook is described as releasing the IPC pointer’s registry entry after cleanup and before user code, replacing the statement that the deleter passes a hook to unlock its std::unique_lock.
observed — Modified behavior in cuda_core/cuda/core/_cpp/rt/REGISTRY_DESIGN.md: Adds documentation of IPC import registries: expired entries persist until the owner releases them after its last driver call, and get_or_create() waits while an expired entry occupies the key. It identifies ipc_pool_registry for imported CUmemoryPool handles and ipc_ptr_cache for CUmemPoolPtrExportData descriptors, with the document’s stated import and cleanup details.
observed — Modified behavior in cuda_core/cuda/core/_cpp/rt/api.hpp: The IPC pool comments replace the automatic-destruction description with documentation that repeated imports share a handle, final release revokes access and destroys once per import, and a racing import waits for release to finish.
observed — Modified behavior in cuda_core/cuda/core/_cpp/rt/api.hpp: Added a public test-hook declaration and comments documenting a delay after an imported pool’s last reference is released and before access revocation; zero disables the delay.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Reviewing files that changed from the base of the PR and between dd7f7af and 8fe6d05.
Files selected for processing: 1
cuda_core/cuda/core/_memory/_memory_pool.pyx
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 9 remain after this review.
suggestion: Reuse the shared child-process runner.
This synchronous wrapper duplicates run_child, but omits its finally cleanup and explicit process-handle closure. Keep the queue transport if needed, and delegate process execution to run_child. The shared runner already handles interrupted waits and closes process handles. (raw.githubusercontent.com)
As per coding guidelines, “Reuse run_child and noop from helpers.child_processes for synchronous child-process tests.”
suggestion: Cover resource closure with an outstanding imported buffer.
This closes the buffer before its resource. Closing first also occurs before buffer import. Neither sequence checks the outstanding-buffer case requested in #3510.
For both pool types, keep an imported buffer alive, close its resource, then use and release the buffer. Capture warnings through final cleanup and assert that no CUDAWarning occurs.
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.
…until NVIDIA#3510 is fixed
Retarget NVIDIA#3516 from NVIDIA#3511 to main. Without NVIDIA#3511, a second import of a pool
in one process gets its own owner of the shared driver pool (NVIDIA#3510).
Dropping a rejected import of the pinned pool then releases that pool and
revokes the device access of the earlier import. Mark that case as a strict
xfail; the XPASS on top of NVIDIA#3511 flags the marker for removal there.
Also drop a doubled blank line in the 1.4.0 release notes left by the rebase.
…ol (NVIDIA#3510)
Every import of a pool in a process returns the same CUmemoryPool and adds a driver reference, while access settings belong to the pool. Releasing one import revoked device access for all of them, which broke a surviving pinned pool import. Imports of a pool now share one owner that revokes access once and destroys every import when the last reference is released; an import racing that release waits for the revocation. Concurrent imports of one UUID return the registered resource instead of raising.
Also cover closing both imports while a buffer imported through one of them
is still alive, as NVIDIA#3510 asks, and run the children through run_child.
…t tests
Use import_pool_resource, pool_access_flags, and is_pool_resource from
helpers.memory, and rely on the device access each import now grants (NVIDIA#3509)
instead of granting it in the test.
…d after closing one import
The concurrent-import test now asserts that dropping the losing imports leaves
the winner's device access intact, which the old per-import ownership broke for
pinned pools. The outstanding-buffer test also checks the shared owner count and
the access left after closing one import.
…ilds
MP_register looked up the UUID and then inserted into the registry as two
separate steps. Without the GIL, two threads importing the same UUID could
both miss and register different resources, contradicting register()'s
documented behavior and failing the concurrent-import test on every Linux
free-threaded CI job. Serialize the lookup and insertion with a lock.
…eRegistry
Replace the hand-rolled IPC pool cache with a HandleRegistry keyed by the
imported CUmemoryPool, as REGISTRY_DESIGN.md describes for raw driver
handles that cuda.core already manages. The owner's box counts the imports
it holds and, when its last reference is released, unregisters, revokes
access once and destroys every import, as before.
Drop the wait for an in-flight release: concurrency.rst makes releasing a
pool while another thread imports it the application's responsibility. A
comment at the registry lookup asks whether a wait is needed there, as in
deviceptr_import_ipc.
Add HandleRegistry::unregister_expired, so that a deleter never erases the
entry of a newer owner registered under the same key.
[P2] Wait for the previous IPC pool owner to finish cleanup before reusing the pool. Once the old owner’s weak reference expires, a concurrent import can register a new owner for the same CUmemoryPool and grant pinned-pool device access. The old deleter can then revoke that access, leaving the new import unusable. This can happen when a buffer drops the last reference to the old owner, without either thread sharing a Python pool object. The previous revision waited through revocation. Restore that ordering and add a regression test that overlaps the last release with a reimport. Pool ownership and cleanup
[P2] Initialize the device ordinal before publishing the resource. Concurrent first imports of one UUID now return the same DeviceMemoryResource, and each caller writes its plain C _dev_id after registration. Those writes race on free-threaded builds even when the IDs agree, and a registry reader can observe CU_DEVICE_INVALID before the first assignment. Resolve and assign the device ID before registration, then reuse the registered resource without rewriting it. Concurrent import return · Device ID assignment
git diff --check passed. memory.cpp passed syntax checks with CUDA 12.9 and 13.5 headers on CPython 3.12 with the GIL and 3.14 free-threaded. The CUDA 12 and 13 driver.pyx and cydriver.pxd wrappers already expose both driver calls.
… before registration
Address review of NVIDIA#3511:
- An import that finds the previous owner's release in flight now waits for
it. The last reference can be dropped by a buffer, so the importer cannot
synchronize with it: the old owner's revocation would remove the access a
new owner granted, and its destroys fail once the new owner has imported a
buffer. ReleasingHandleRegistry, a HandleRegistry variant whose entry
outlives its handle until the owner's deleter calls release(), provides the
wait. The pool deleter revokes access and destroys every import before it
releases the entry, and reports a failed destroy only afterwards.
- The IPC pointer cache uses the same registry instead of its own mutex and
condition variable. Its deleter releases the entry from cleanup_in_context's
after_cleanup hook, once the mapping is freed.
- DeviceMemoryResource.from_allocation_handle resolves the device before the
import and sets it in the import hook, before registration. A concurrent
import of the same UUID returns the registered resource without writing to
it.
Add a test that races a re-import against a buffer dropping the last
reference to the pool, and check device_id in the concurrent UUID test.
…hook
A re-import rarely lands between an owner's last reference and its cleanup,
so the race test passed without the wait as well. Add
_set_ipc_pool_release_delay_for_testing, which makes the owner of an
imported pool sleep after its last reference is released and before it
revokes access. With a 50 ms delay, the test re-imports inside the release
and checks the new import after it: without the wait, device pools report
failed cuMemPoolDestroy calls and pinned pools lose device access.
Unverified finding
Verification ran but could not confirm this finding. It is shown for review, not as a verified issue.
important: Use an explicit stream for the cross-thread race.
With CUDA_PYTHON_CUDA_PER_THREAD_DEFAULT_STREAM enabled, the main thread records its per-thread default stream on outstanding. The worker closes that buffer using the recorded stream. The runtime reports a cross-thread PTDS warning, so assert_no_cuda_warning() can fail even when pool cleanup works. Use an explicitly created stream and keep it alive through worker completion and buffer cleanup. (raw.githubusercontent.com)
Files skipped from review as they are similar to previous changes: 2
cuda_core/cuda/core/_cpp/rt/REGISTRY_DESIGN.md
cuda_core/docs/source/release/1.4.0-notes.rst
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.
…ate, and address review nits
The pointer-import factory was noexcept but calls make_deallocation_stream,
which may throw std::bad_alloc. Drop noexcept; on a throw, discard the import
under the cache's mutex and report a failed discard after it is released.
Also report how many cuMemPoolDestroy calls failed when an IPC pool owner held
several imports, abort the race tests' barriers when a worker fails, and use
an explicit stream for the buffer freed on another thread.
[P3] Make UUID registration safe for same-thread reentry. The new non-reentrant lock spans WeakValueDictionary insertion. Allocating the weak reference can run a cyclic-GC finalizer while the lock is held. If that finalizer registers another IPC resource, it blocks on the same lock. Keep winner selection atomic while allowing callbacks to reenter. Registry insertion
Both earlier findings are resolved in the code. A new owner waits for the previous owner’s revocation and destroy calls, and _dev_id is set before registration.
All nine failed Linux jobs I checked had the same sole test failure. The new pinned-pool race test hit CUDA_ERROR_ALREADY_MAPPED in cuMemPoolSetAccess, matching #3514. One failing test repeated across the CI matrix accounts for the widespread failures I saw. CUDA 12 job · CUDA 13 job · #3514
The PR now retries the pinned-pool access grant once on that exact error. That targets the observed failure. CI on the new head is still pending. Follow-up commit
…_ALREADY_MAPPED
On drivers 580 and 595, re-importing a pinned pool whose previous owner has
revoked access and destroyed its import, while a free of an imported pointer
was not synchronized, returns the same CUmemoryPool and fails the first
cuMemPoolSetAccess with CUDA_ERROR_ALREADY_MAPPED. An identical retry succeeds.
Retry once, on that status only and only on the import path.
Inserting into the registry allocates a weak reference, which can start a
garbage collection. A callback or finalizer that registers another IPC
resource then ran on the thread holding the non-reentrant lock and deadlocked.
Use an RLock, and add a test that registers from a GC callback.
The reason will be displayed to describe this comment to others. Learn more.
codex conversation:
codex:
One narrower P3 remains. If the callback registers a different resource under the same UUID after the outer call has missed the lookup, the inner call can publish and return its resource. The outer call then overwrites that entry and returns a different resource. I reproduced this ordering with an ordinary UUID, real cyclic GC, and a CPU-only model of the registry code. The new CUDA test reenters with a different UUID that is already registered, so it covers the original deadlock. A follow-on could make insertion select one winner even when a callback registers the key during preparation. Registration code · GC regression test
rwgk:
Do you think this could lead to flakiness?
codex:
Yes, in a narrow case. If a GC callback registers a different resource under the same UUID, the outcome depends on when GC runs. If it runs between the outer call’s lookup and insertion, both calls can return different resources as the winner. If the outer resource is then collected, a later from_registry(uuid) can fail even though the inner resource is still alive.
I’d treat that as possible application flakiness. The current regression test uses a different, already-registered UUID, so this particular bug shouldn’t make that test flaky.
I highlighted application flakiness manually. It seems like something we should fix in a follow-on?
… import grant
With two devices, one retry of the combined cuMemPoolSetAccess call still
failed with CUDA_ERROR_ALREADY_MAPPED on driver 595. Set each entry on its
own and retry each once.
codex is suggesting hardening the new test some more:
Reviewed #3511 at dc522f7. The new commit targets the remaining two-GPU failure. I verified that the parent revision still failed the pinned reimport test with CUDA_ERROR_ALREADY_MAPPED on both dual-GPU jobs running driver 595.99.02. The new code grants access one GPU at a time, retries that exact error once per GPU, and propagates any remaining failure. I found no correctness issue in the new loop. Prior T4 job · Prior H100 job · New code
[P3] Check every supported GPU after the reimport. The race test checks access and writes a buffer only on the default GPU. It catches the old batched retry raising, but it would pass if a reimport silently left GPU 1 without access. The separate second-GPU test covers a fresh import. Checking read/write access on every supported GPU after the race would cover the behavior this commit changes. Race test · Fresh-import test
Thanks! We'll leave both open P3s for a follow-up PR (#3524) rather than add another CI round here, since #3516 is waiting on this one:
Check every supported GPU after the reimport race. Agreed that the race test should assert read/write access and use the buffer on every GPU that supports the pool, not only the default one. On dc522f7 both dual-GPU jobs that failed before (T4 and H100) now pass, so the per-GPU grant no longer raises there; the follow-up adds the check that no GPU is silently left without access.
Same-UUID reentry in MP_register. The overwrite also exists on main, which registers with no lock at all. Fixing it means making sure that, of the two registrations, only one wins, even when a callback registers the same key during insertion. That likely means replacing the WeakValueDictionary with weak references we create before the check, which touches the registry's lookup paths too.
…ol (NVIDIA#3511, squashed)
Temporary squash of NVIDIA#3511 at dc522f7 onto main, standing in for its
merge so that NVIDIA#3516 can be prepared on top. Not for pushing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bugSomething isn't workingcuda.coreEverything related to the cuda.core module
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
closes #3510
Milestone: cuda.core 1.4.0 (release note in
cuda_core/docs/source/release/1.4.0-notes.rst).Importing the same memory pool more than once in one process (for example
PinnedMemoryResource.from_allocation_handle(int(fd))twice, or two threads importing the same UUID) gave independent owners of one driver pool. Closing one of them broke the others for pinned pools, and concurrent imports of one UUID raised "internal cuda.core error ... register() returned a different memory resource".Root cause
Every import of a pool in a process returns the same
CUmemoryPool, and the driver counts the imports. Butcreate_mempool_handle_ipcwrapped each import in its own resource handle, so one pool had two reference counts. Closing one handle ran the pool deleter's access revocation (the nvbug 5698116 workaround), and because access settings belong to the pool, that removed the device access the other import relied on. Verified with rawcuda.bindingscalls on driver 595.99.02.Fix
create_mempool_handle_ipcfinds an existing owner through a registry keyed by the importedCUmemoryPool(Level 1 inREGISTRY_DESIGN.md). A repeat import adds one to the owner's import count and returns its handle. When the last reference is released, the owner revokes access once and then callscuMemPoolDestroyonce per import: destroying a duplicate import early fails withCUDA_ERROR_INVALID_VALUEwhile buffers imported from the pool are outstanding.ReleasingHandleRegistry, aHandleRegistryvariant whose entry outlives its handle until the owner's deleter callsrelease(), provides the wait. The deleter revokes access and destroys every import before it releases the entry.deviceptr_import_ipc(the nvbug 5570902 workaround) uses the same registry instead of its own mutex and condition variable.MP_from_allocation_handlereturns the registered resource when a concurrent import of the same UUID registered first, instead of raising.MP_registerholds a lock across its registry lookup and insertion, because on free-threaded builds two threads could both miss and register different resources for one UUID.DeviceMemoryResource.from_allocation_handlesets the device in the import hook, before registration, so a concurrent import never writes to a registered resource. A registry hit keeps the resource's originaldevice_idinstead of changing it to the requested device.Tests
New
cuda_core/tests/memory_ipc/test_ipc_pool_duplicate_import.py, for device and pinned pools:CUDAWarning.CUDAWarning. A test hook,_set_ipc_pool_release_delay_for_testing, delays the old owner's cleanup so the overlap is deterministic. Without the wait, the device pool case reports failedcuMemPoolDestroycalls and the pinned case loses device access.Against unfixed
main, the pinned close test and the concurrent test also fail.Notes for reviewers
cuMemPoolDestroymatches a TODO in the driver'scuapiMemPoolDestroy("Relax this and allow destruction to succeed even if memobj exists"). A pendingcuMemFreeAsyncof an imported pointer does not block it.cuda_coreis affected; neither bindings root has a counterpart.Checklist