Skip to content

fix(cuda.core): share ownership of repeat imports of an IPC memory pool - #3511

Merged
juenglin merged 13 commits into
NVIDIA:mainfrom
juenglin:fix-mempool-import-refcount
Nov 21, 2026
Merged

juenglin merged 13 commits into
NVIDIA:mainfrom
juenglin:fix-mempool-import-refcount

Conversation

@juenglin

@juenglin juenglin commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Description

closes #3510

Milestone: cuda.core 1.4.0 (release note in cuda_core/docs/source/release/1.4.0-notes.rst).

Importing the same memory pool more than once in one process (for example PinnedMemoryResource.from_allocation_handle(int(fd)) twice, or two threads importing the same UUID) gave independent owners of one driver pool. Closing one of them broke the others for pinned pools, and concurrent imports of one UUID raised "internal cuda.core error ... register() returned a different memory resource".

Root cause

Every import of a pool in a process returns the same CUmemoryPool, and the driver counts the imports. But create_mempool_handle_ipc wrapped each import in its own resource handle, so one pool had two reference counts. Closing one handle ran the pool deleter's access revocation (the nvbug 5698116 workaround), and because access settings belong to the pool, that removed the device access the other import relied on. Verified with raw cuda.bindings calls on driver 595.99.02.

Fix

  • create_mempool_handle_ipc finds an existing owner through a registry keyed by the imported CUmemoryPool (Level 1 in REGISTRY_DESIGN.md). A repeat import adds one to the owner's import count and returns its handle. When the last reference is released, the owner revokes access once and then calls cuMemPoolDestroy once per import: destroying a duplicate import early fails with CUDA_ERROR_INVALID_VALUE while buffers imported from the pool are outstanding.
  • An import that finds the previous owner's release in flight waits for it. The last reference can be dropped by an imported buffer or by garbage collection, so the application cannot synchronize with it. Without the wait, the old owner's revocation removes the access a new owner granted, and its destroys fail once the new owner has imported a buffer. ReleasingHandleRegistry, a HandleRegistry variant whose entry outlives its handle until the owner's deleter calls release(), provides the wait. The deleter revokes access and destroys every import before it releases the entry.
  • The IPC pointer cache in deviceptr_import_ipc (the nvbug 5570902 workaround) uses the same registry instead of its own mutex and condition variable.
  • MP_from_allocation_handle returns the registered resource when a concurrent import of the same UUID registered first, instead of raising. MP_register holds a lock across its registry lookup and insertion, because on free-threaded builds two threads could both miss and register different resources for one UUID.
  • DeviceMemoryResource.from_allocation_handle sets the device in the import hook, before registration, so a concurrent import never writes to a registered resource. A registry hit keeps the resource's original device_id instead of changing it to the requested device.

Tests

New cuda_core/tests/memory_ipc/test_ipc_pool_duplicate_import.py, for device and pinned pools:

  • Closing one of two imports keeps the other usable.
  • A buffer imported through one import survives closing both, with no CUDAWarning.
  • Concurrent imports of one UUID return one resource, with its device set, and one pool.
  • A re-import that overlaps the release of the last import keeps device access and emits no CUDAWarning. A test hook, _set_ipc_pool_release_delay_for_testing, delays the old owner's cleanup so the overlap is deterministic. Without the wait, the device pool case reports failed cuMemPoolDestroy calls and the pinned case loses device access.

Against unfixed main, the pinned close test and the concurrent test also fail.

Notes for reviewers

  • The deleter releases the registry entry only after its destroys, not after the revocation: a woken importer could otherwise import a buffer before the old owner's destroys run, and they would fail.
  • The refused cuMemPoolDestroy matches a TODO in the driver's cuapiMemPoolDestroy ("Relax this and allow destruction to succeed even if memobj exists"). A pending cuMemFreeAsync of an imported pointer does not block it.
  • Peer access with several GPUs takes the same path but was tested on a single GPU only.
  • Only cuda_core is affected; neither bindings root has a counterpart.

Checklist

  • I have read the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@juenglin juenglin added this to the cuda.core 1.4.0 milestone Nov 20, 2026
@juenglin juenglin added bug Something isn't working cuda.core Everything related to the cuda.core module labels Nov 20, 2026
@juenglin juenglin self-assigned this Nov 20, 2026
@copy-pr-bot

copy-pr-bot Bot commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Explain Change →

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • Resume reviews
  • Trigger review

No actionable comments were generated in the recent review.

Summary

Summary by CodeRabbit

  • Bug Fixes
    • Repeated imports of the same memory pool now share ownership. Closing one import preserves access through the others; access is revoked when the final import is released.
    • Outstanding buffers keep their memory pool available after its imports are closed.
    • Concurrent imports of the same allocation handle now return the same registered resource.
    • Imports that overlap with a pool’s final release wait for cleanup to complete before proceeding.
    • When an allocation handle is already registered, the returned resource retains its original device.

Repeated IPC memory-pool imports now share ownership until final release. The pointer cache waits for cleanup before re-importing a handle. Concurrent UUID imports return the registered resource after a type check. Tests cover duplicate imports, outstanding buffers, release races, and concurrent UUID imports.

Changes: IPC memory-pool imports

Layer / File(s) Summary
Shared pool ownership and release
cuda_core/cuda/core/_cpp/rt/api.hpp, cuda_core/cuda/core/_cpp/rt/memory.cpp, cuda_core/cuda/core/_memory/_memory_pool.pxd, cuda_core/cuda/core/_memory/_memory_pool.pyx, cuda_core/cuda/core/_memory/_pinned_memory_resource.pyx, cuda_core/cuda/core/_rt.pyx, cuda_core/cuda/core/_rt.pyi, cuda_core/tests/memory_ipc/test_ipc_pool_duplicate_import.py, cuda_core/docs/source/release/1.4.0-notes.rst
The C++ import path shares ownership for repeated pool handles and counts driver imports. Final release revokes access and destroys each import. The release-delay test hook and imported-pool access retry support the release path. Tests cover duplicate imports, outstanding buffers, and release races; release notes describe the behavior.
UUID registry imports and initialization
cuda_core/cuda/core/_memory/_ipc.pyx, cuda_core/cuda/core/_memory/_ipc.pxd, cuda_core/cuda/core/_memory/_ipc.pyi, cuda_core/cuda/core/_memory/_device_memory_resource.pyx, cuda_core/cuda/core/_memory/_device_memory_resource.pyi, cuda_core/tests/memory_ipc/test_ipc_pool_duplicate_import.py, cuda_core/docs/source/release/1.4.0-notes.rst
The Python registry serializes UUID lookup and insertion, checks the registered resource type, and returns the existing resource if another import registers first. Initialization occurs before registration. An existing device resource retains its original device. Tests check concurrent imports and device IDs.
IPC pointer import cleanup
cuda_core/cuda/core/_cpp/rt/memory.cpp, cuda_core/cuda/core/_cpp/rt/internal.hpp, cuda_core/cuda/core/_cpp/rt/DESIGN.md, cuda_core/cuda/core/_cpp/rt/REGISTRY_DESIGN.md, cuda_core/tests/test_rt_layout.py
The pointer cache uses a releasing-handle registry. An import waits while prior mapping cleanup is in progress. The design documentation describes when the registry entry is released.

Assessment against linked issues

Objective Addressed Explanation
Share ownership of repeated imports of the same memory pool for pinned and device resources, including while imported buffers remain open [#3510] ✅
Return the already-registered resource when concurrent imports of the same UUID race, after checking its type [#3510] ✅

Possibly related issues

  • NVIDIA/cuda-python#3514: The issue reports a remaining imported-pinned-pool release failure with imported pointers outstanding and states that it reproduces with this change.

Change: Bug fix · Severity of issue fixed: Medium

Merge Risk: Low · previous review 8fe6d

Repeated imports of an IPC memory pool now share ownership, which fixes premature access revocation. Remaining concerns are narrow: a deadlock under an unusual warning-handler re-import, a possible abort on an allocation failure, and a timing-based test that could pass without exercising the race. These are acceptable to follow up on, so merge risk is low.

Architecture risk: Low · up to 8fe6d

The change affects 1 system.

Changed systems: cuda_core

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

  • observed — cuda_core (service) was modified; 18 changed files map to changed impact.

Before / after behavior

  • observed — Modified behavior in cuda_core/cuda/core/_cpp/rt/DESIGN.md: The GIL-locking rules remain, but the deviceptr_import_ipc example now describes reporting a discarded import after releasing the lock. The after_cleanup hook is described as releasing the IPC pointer’s registry entry after cleanup and before user code, replacing the statement that the deleter passes a hook to unlock its std::unique_lock.
  • observed — Modified behavior in cuda_core/cuda/core/_cpp/rt/REGISTRY_DESIGN.md: Adds documentation of IPC import registries: expired entries persist until the owner releases them after its last driver call, and get_or_create() waits while an expired entry occupies the key. It identifies ipc_pool_registry for imported CUmemoryPool handles and ipc_ptr_cache for CUmemPoolPtrExportData descriptors, with the document’s stated import and cleanup details.
  • observed — Modified behavior in cuda_core/cuda/core/_cpp/rt/api.hpp: The IPC pool comments replace the automatic-destruction description with documentation that repeated imports share a handle, final release revokes access and destroys once per import, and a racing import waits for release to finish.
  • observed — Modified behavior in cuda_core/cuda/core/_cpp/rt/api.hpp: Added a public test-hook declaration and comments documenting a delay after an imported pool’s last reference is released and before access revocation; zero disables the delay.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between dd7f7af and 8fe6d05.

Files selected for processing: 1

  • cuda_core/cuda/core/_memory/_memory_pool.pyx

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 9 remain after this review.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Nitpick comments: 2

cuda_core/tests/memory_ipc/test_ipc_pool_duplicate_import.py: 2

132-132: Trivial · Maintainability & Code Quality · Quick win

suggestion: Reuse the shared child-process runner.

This synchronous wrapper duplicates run_child, but omits its finally cleanup and explicit process-handle closure. Keep the queue transport if needed, and delegate process execution to run_child. The shared runner already handles interrupted waits and closes process handles. (raw.githubusercontent.com)

As per coding guidelines, “Reuse run_child and noop from helpers.child_processes for synchronous child-process tests.”

Source: Coding guidelines


86-87: Trivial · Maintainability & Code Quality · Quick win

suggestion: Cover resource closure with an outstanding imported buffer.

This closes the buffer before its resource. Closing first also occurs before buffer import. Neither sequence checks the outstanding-buffer case requested in #3510.

For both pool types, keep an imported buffer alive, close its resource, then use and release the buffer. Capture warnings through final cleanup and assert that no CUDAWarning occurs.

Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 0a21370 and 719858b.

Files selected for processing: 6

  • cuda_core/cuda/core/_cpp/rt/api.hpp
  • cuda_core/cuda/core/_cpp/rt/memory.cpp
  • cuda_core/cuda/core/_memory/_ipc.pyx
  • cuda_core/docs/nv-versions.json
  • cuda_core/docs/source/release/1.4.0-notes.rst
  • cuda_core/tests/memory_ipc/test_ipc_pool_duplicate_import.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

Comment thread cuda_core/cuda/core/_cpp/rt/memory.cpp Resolved
@juenglin
juenglin force-pushed the fix-mempool-import-refcount branch from 719858b to 3984b7b Compare November 20, 2026 01:34
@juenglin
juenglin requested a review from Andy-Jost November 20, 2026 01:39
@juenglin

Copy link
Copy Markdown
Contributor Author

/ok to test c9af736

@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor
Doc Preview CI
Preview removed because the pull request was closed or merged.

@juenglin

Copy link
Copy Markdown
Contributor Author

/ok to test 6c7a7c9

@juenglin
juenglin requested a review from rwgk November 20, 2026 18:54
juenglin added a commit to juenglin/cuda-python that referenced this pull request Nov 20, 2026
…until NVIDIA#3510 is fixed

Retarget NVIDIA#3516 from NVIDIA#3511 to main. Without NVIDIA#3511, a second import of a pool
in one process gets its own owner of the shared driver pool (NVIDIA#3510).
Dropping a rejected import of the pinned pool then releases that pool and
revokes the device access of the earlier import. Mark that case as a strict
xfail; the XPASS on top of NVIDIA#3511 flags the marker for removal there.

Also drop a doubled blank line in the 1.4.0 release notes left by the rebase.
…ol (NVIDIA#3510)

Every import of a pool in a process returns the same CUmemoryPool and adds a driver reference, while access settings belong to the pool. Releasing one import revoked device access for all of them, which broke a surviving pinned pool import. Imports of a pool now share one owner that revokes access once and destroys every import when the last reference is released; an import racing that release waits for the revocation. Concurrent imports of one UUID return the registered resource instead of raising.
Also cover closing both imports while a buffer imported through one of them
is still alive, as NVIDIA#3510 asks, and run the children through run_child.
…t tests

Use import_pool_resource, pool_access_flags, and is_pool_resource from
helpers.memory, and rely on the device access each import now grants (NVIDIA#3509)
instead of granting it in the test.
…d after closing one import

The concurrent-import test now asserts that dropping the losing imports leaves
the winner's device access intact, which the old per-import ownership broke for
pinned pools. The outstanding-buffer test also checks the shared owner count and
the access left after closing one import.
…ilds

MP_register looked up the UUID and then inserted into the registry as two
separate steps. Without the GIL, two threads importing the same UUID could
both miss and register different resources, contradicting register()'s
documented behavior and failing the concurrent-import test on every Linux
free-threaded CI job. Serialize the lookup and insertion with a lock.
…eRegistry

Replace the hand-rolled IPC pool cache with a HandleRegistry keyed by the
imported CUmemoryPool, as REGISTRY_DESIGN.md describes for raw driver
handles that cuda.core already manages. The owner's box counts the imports
it holds and, when its last reference is released, unregisters, revokes
access once and destroys every import, as before.

Drop the wait for an in-flight release: concurrency.rst makes releasing a
pool while another thread imports it the application's responsibility. A
comment at the registry lookup asks whether a wait is needed there, as in
deviceptr_import_ipc.

Add HandleRegistry::unregister_expired, so that a deleter never erases the
entry of a newer owner registered under the same key.
@juenglin
juenglin force-pushed the fix-mempool-import-refcount branch from 6c7a7c9 to ec5a3a9 Compare November 20, 2026 22:23
@juenglin

Copy link
Copy Markdown
Contributor Author

/ok to test ec5a3a9

@rwgk rwgk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

codex GPT-6.2-Sol ultra:

Two findings in at ec5a3a9:

  • [P2] Wait for the previous IPC pool owner to finish cleanup before reusing the pool. Once the old owner’s weak reference expires, a concurrent import can register a new owner for the same CUmemoryPool and grant pinned-pool device access. The old deleter can then revoke that access, leaving the new import unusable. This can happen when a buffer drops the last reference to the old owner, without either thread sharing a Python pool object. The previous revision waited through revocation. Restore that ordering and add a regression test that overlaps the last release with a reimport. Pool ownership and cleanup
  • [P2] Initialize the device ordinal before publishing the resource. Concurrent first imports of one UUID now return the same DeviceMemoryResource, and each caller writes its plain C _dev_id after registration. Those writes race on free-threaded builds even when the IDs agree, and a registry reader can observe CU_DEVICE_INVALID before the first assignment. Resolve and assign the device ID before registration, then reuse the registered resource without rewriting it. Concurrent import return · Device ID assignment

git diff --check passed. memory.cpp passed syntax checks with CUDA 12.9 and 13.5 headers on CPython 3.12 with the GIL and 3.14 free-threaded. The CUDA 12 and 13 driver.pyx and cydriver.pxd wrappers already expose both driver calls.

@rwgk rwgk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving, assuming the two codex reviews I posted here and on #3516 will be addressed.

… before registration

Address review of NVIDIA#3511:

- An import that finds the previous owner's release in flight now waits for
  it. The last reference can be dropped by a buffer, so the importer cannot
  synchronize with it: the old owner's revocation would remove the access a
  new owner granted, and its destroys fail once the new owner has imported a
  buffer. ReleasingHandleRegistry, a HandleRegistry variant whose entry
  outlives its handle until the owner's deleter calls release(), provides the
  wait. The pool deleter revokes access and destroys every import before it
  releases the entry, and reports a failed destroy only afterwards.
- The IPC pointer cache uses the same registry instead of its own mutex and
  condition variable. Its deleter releases the entry from cleanup_in_context's
  after_cleanup hook, once the mapping is freed.
- DeviceMemoryResource.from_allocation_handle resolves the device before the
  import and sets it in the import hook, before registration. A concurrent
  import of the same UUID returns the registered resource without writing to
  it.

Add a test that races a re-import against a buffer dropping the last
reference to the pool, and check device_id in the concurrent UUID test.
…hook

A re-import rarely lands between an owner's last reference and its cleanup,
so the race test passed without the wait as well. Add
_set_ipc_pool_release_delay_for_testing, which makes the owner of an
imported pool sleep after its last reference is released and before it
revokes access. With a 50 ms delay, the test re-imports inside the release
and checks the new import after it: without the wait, device pools report
failed cuMemPoolDestroy calls and pinned pools lose device access.
@juenglin

Copy link
Copy Markdown
Contributor Author

/ok to test 6b11e5e

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

Additional comments: 1

cuda_core/tests/memory_ipc/test_ipc_pool_duplicate_import.py-99-99: 1

99-99: Minor · Functional Correctness · Quick win

Unverified finding
Verification ran but could not confirm this finding. It is shown for review, not as a verified issue.

important: Use an explicit stream for the cross-thread race.

With CUDA_PYTHON_CUDA_PER_THREAD_DEFAULT_STREAM enabled, the main thread records its per-thread default stream on outstanding. The worker closes that buffer using the recorded stream. The runtime reports a cross-thread PTDS warning, so assert_no_cuda_warning() can fail even when pool cleanup works. Use an explicitly created stream and keep it alive through worker completion and buffer cleanup. (raw.githubusercontent.com)

Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between ec5a3a9 and 6b11e5e.

Files selected for processing: 16

  • cuda_core/cuda/core/_cpp/rt/DESIGN.md
  • cuda_core/cuda/core/_cpp/rt/REGISTRY_DESIGN.md
  • cuda_core/cuda/core/_cpp/rt/api.hpp
  • cuda_core/cuda/core/_cpp/rt/internal.hpp
  • cuda_core/cuda/core/_cpp/rt/memory.cpp
  • cuda_core/cuda/core/_memory/_device_memory_resource.pyi
  • cuda_core/cuda/core/_memory/_device_memory_resource.pyx
  • cuda_core/cuda/core/_memory/_ipc.pxd
  • cuda_core/cuda/core/_memory/_ipc.pyi
  • cuda_core/cuda/core/_memory/_ipc.pyx
  • cuda_core/cuda/core/_memory/_pinned_memory_resource.pyx
  • cuda_core/cuda/core/_rt.pyi
  • cuda_core/cuda/core/_rt.pyx
  • cuda_core/docs/source/release/1.4.0-notes.rst
  • cuda_core/tests/memory_ipc/test_ipc_pool_duplicate_import.py
  • cuda_core/tests/test_rt_layout.py
Files skipped from review as they are similar to previous changes: 2

  • cuda_core/cuda/core/_cpp/rt/REGISTRY_DESIGN.md
  • cuda_core/docs/source/release/1.4.0-notes.rst

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

Comment thread cuda_core/cuda/core/_cpp/rt/memory.cpp Resolved
Comment thread cuda_core/cuda/core/_cpp/rt/memory.cpp Resolved
Comment thread cuda_core/cuda/core/_cpp/rt/memory.cpp Resolved
…ate, and address review nits

The pointer-import factory was noexcept but calls make_deallocation_stream,
which may throw std::bad_alloc. Drop noexcept; on a throw, discard the import
under the cache's mutex and report a failed discard after it is released.

Also report how many cuMemPoolDestroy calls failed when an IPC pool owner held
several imports, abort the race tests' barriers when a worker fails, and use
an explicit stream for the buffer freed on another thread.
@juenglin

Copy link
Copy Markdown
Contributor Author

/ok to test c098961

@juenglin
juenglin enabled auto-merge (squash) November 20, 2026 23:47
@juenglin

Copy link
Copy Markdown
Contributor Author

/ok to test dd7f7af

@rwgk

rwgk commented Nov 21, 2026

Copy link
Copy Markdown
Contributor

Stitching a couple codex responses together:

Re-reviewed PR #3511 at HEAD c098961.

  • [P1] Drain the buffer free before racing the last pool release. The new pinned-pool test fails in both CUDA 12 and CUDA 13 CI with CUDA_ERROR_ALREADY_MAPPED in cuMemPoolSetAccess. Closing outstanding queues an asynchronous free and immediately drops the last pool reference. That exercises the pending-free bug tracked in [BUG]: Re-importing a PinnedMemoryResource pool fails with CUDA_ERROR_ALREADY_MAPPED after a release with a pending buffer free #3514 even when the registry wait works. Keep the pool owner alive through outstanding.close() and stream.sync(), then race its final release against the reimport. Race test · CUDA 12 CI · CUDA 13 CI · #3514
  • [P3] Make UUID registration safe for same-thread reentry. The new non-reentrant lock spans WeakValueDictionary insertion. Allocating the weak reference can run a cyclic-GC finalizer while the lock is held. If that finalizer registers another IPC resource, it blocks on the same lock. Keep winner selection atomic while allowing callbacks to reenter. Registry insertion

Both earlier findings are resolved in the code. A new owner waits for the previous owner’s revocation and destroy calls, and _dev_id is set before registration.


All nine failed Linux jobs I checked had the same sole test failure. The new pinned-pool race test hit CUDA_ERROR_ALREADY_MAPPED in cuMemPoolSetAccess, matching #3514. One failing test repeated across the CI matrix accounts for the widespread failures I saw. CUDA 12 job · CUDA 13 job · #3514

The PR now retries the pinned-pool access grant once on that exact error. That targets the observed failure. CI on the new head is still pending. Follow-up commit

@rwgk

rwgk commented Nov 21, 2026

Copy link
Copy Markdown
Contributor

Lines crossed :-)

…_ALREADY_MAPPED

On drivers 580 and 595, re-importing a pinned pool whose previous owner has
revoked access and destroyed its import, while a free of an imported pointer
was not synchronized, returns the same CUmemoryPool and fails the first
cuMemPoolSetAccess with CUDA_ERROR_ALREADY_MAPPED. An identical retry succeeds.
Retry once, on that status only and only on the import path.
@juenglin
juenglin force-pushed the fix-mempool-import-refcount branch from dd7f7af to 8fe6d05 Compare November 21, 2026 00:27
Inserting into the registry allocates a weak reference, which can start a
garbage collection. A callback or finalizer that registers another IPC
resource then ran on the thread holding the non-reentrant lock and deadlocked.
Use an RLock, and add a test that registers from a GC callback.
@juenglin

Copy link
Copy Markdown
Contributor Author

/ok to test 86b2ac8

@rwgk rwgk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

codex conversation:

codex:

One narrower P3 remains. If the callback registers a different resource under the same UUID after the outer call has missed the lookup, the inner call can publish and return its resource. The outer call then overwrites that entry and returns a different resource. I reproduced this ordering with an ordinary UUID, real cyclic GC, and a CPU-only model of the registry code. The new CUDA test reenters with a different UUID that is already registered, so it covers the original deadlock. A follow-on could make insertion select one winner even when a callback registers the key during preparation. Registration code · GC regression test

rwgk:

Do you think this could lead to flakiness?

codex:

Yes, in a narrow case. If a GC callback registers a different resource under the same UUID, the outcome depends on when GC runs. If it runs between the outer call’s lookup and insertion, both calls can return different resources as the winner. If the outer resource is then collected, a later from_registry(uuid) can fail even though the inner resource is still alive.

I’d treat that as possible application flakiness. The current regression test uses a different, already-registered UUID, so this particular bug shouldn’t make that test flaky.


I highlighted application flakiness manually. It seems like something we should fix in a follow-on?

… import grant

With two devices, one retry of the combined cuMemPoolSetAccess call still
failed with CUDA_ERROR_ALREADY_MAPPED on driver 595. Set each entry on its
own and retry each once.
@juenglin

Copy link
Copy Markdown
Contributor Author

/ok to test dc522f7

@rwgk

rwgk commented Nov 21, 2026

Copy link
Copy Markdown
Contributor

codex is suggesting hardening the new test some more:

Reviewed #3511 at dc522f7. The new commit targets the remaining two-GPU failure. I verified that the parent revision still failed the pinned reimport test with CUDA_ERROR_ALREADY_MAPPED on both dual-GPU jobs running driver 595.99.02. The new code grants access one GPU at a time, retries that exact error once per GPU, and propagates any remaining failure. I found no correctness issue in the new loop. Prior T4 job · Prior H100 job · New code

  • [P3] Check every supported GPU after the reimport. The race test checks access and writes a buffer only on the default GPU. It catches the old batched retry raising, but it would pass if a reimport silently left GPU 1 without access. The separate second-GPU test covers a fresh import. Checking read/write access on every supported GPU after the race would cover the behavior this commit changes. Race test · Fresh-import test

The earlier same-UUID reentry P3 remains open.

@juenglin

juenglin commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks! We'll leave both open P3s for a follow-up PR (#3524) rather than add another CI round here, since #3516 is waiting on this one:

  • Check every supported GPU after the reimport race. Agreed that the race test should assert read/write access and use the buffer on every GPU that supports the pool, not only the default one. On dc522f7 both dual-GPU jobs that failed before (T4 and H100) now pass, so the per-GPU grant no longer raises there; the follow-up adds the check that no GPU is silently left without access.
  • Same-UUID reentry in MP_register. The overwrite also exists on main, which registers with no lock at all. Fixing it means making sure that, of the two registrations, only one wins, even when a callback registers the same key during insertion. That likely means replacing the WeakValueDictionary with weak references we create before the check, which touches the registry's lookup paths too.

Update: both items are addressed in #3524.

juenglin added a commit to juenglin/cuda-python that referenced this pull request Nov 21, 2026
…ol (NVIDIA#3511, squashed)

Temporary squash of NVIDIA#3511 at dc522f7 onto main, standing in for its
merge so that NVIDIA#3516 can be prepared on top. Not for pushing.
@juenglin
juenglin merged commit 7de038b into NVIDIA:main Nov 21, 2026
223 of 226 checks passed
github-actions Bot pushed a commit that referenced this pull request Nov 21, 2026
Removed preview folders for the following PRs:
- PR #2924
- PR #3470
- PR #3490
- PR #3511
- PR #3515
- PR #3516
- PR #3520
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working cuda.core Everything related to the cuda.core module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: Importing a memory pool twice in one process creates two owners of one driver pool

2 participants