Skip to content

[BUG]: Importing a memory pool twice in one process creates two owners of one driver pool #3510

Description

@juenglin

Is this a duplicate?

Type of Bug

Runtime Error

Component

cuda.core

Describe the bug

Importing the same memory pool twice in one process gives two memory resource objects that each own the same driver pool. This affects both PinnedMemoryResource and DeviceMemoryResource, because both import through the same code path: MP_from_allocation_handle in cuda_core/cuda/core/_memory/_ipc.pyx calls create_mempool_handle_ipc in cuda_core/cuda/core/_cpp/rt/memory.cpp.

cuMemPoolImportFromShareableHandle returns the same CUmemoryPool handle when a process imports the same pool again, and the driver counts the imports: each one needs its own cuMemPoolDestroy, and the pool survives until the last of them. create_mempool_handle_ipc wraps every import as a separate owning handle, so the destroys themselves are balanced. What breaks is the cleanup that each owner runs before its destroy: it revokes device access to the pool (the nvbug 5698116 workaround in clear_mempool_peer_access). Access settings belong to the pool, not to an import, so one owner's revocation removes access that the other import still relies on.

How this shows up depends on the pool type:

  • Pinned (host) pools: a host pool is accessible only from the host unless access is granted, so closing one resource revokes the device access the other one relies on. Importing a buffer through the surviving resource then fails with CUDA_ERROR_INVALID_VALUE.
  • Device pools: a device pool stays accessible from its own device, so on a single GPU the revocation is harmless and the surviving resource keeps working. (Peer access granted to other devices is still revoked.) The visible failure is a different one: the driver refuses cuMemPoolDestroy while pointers imported from the pool are outstanding. If a buffer imported through one resource is still alive when the other resource is closed, that destroy fails with CUDA_ERROR_INVALID_VALUE (reported as a CUDAWarning), and the import's driver reference is not released.

Both behaviors were confirmed with raw cuda.bindings calls: after two imports of a pinned pool, one cuMemPoolDestroy leaves the other import with read/write device access, while revoking access before that destroy leaves it with none. As #3511 notes, the refused destroy matches a TODO in the driver's cuapiMemPoolDestroy ("Relax this and allow destruction to succeed even if memobj exists"), so it is current driver behavior rather than a bug for cuda.core to report.

The registry hides the double ownership on most paths, because a second import of a handle with a UUID returns the registered resource without importing again. The bug is reachable in two ways, for both resource types:

  • Raw handles: PinnedMemoryResource.from_allocation_handle(int(fd)) and DeviceMemoryResource.from_allocation_handle(device, int(fd)) carry no UUID, so each call imports the pool again. This path is deterministic and needs no concurrency.
  • Concurrent imports of the same UUID: two threads can both miss the registry and both import. MP_from_allocation_handle then raises "internal cuda.core error, please report: register() returned a different memory resource than the one being registered" in the thread that registered second. Dropping that thread's resource also revokes device access to the pool that the other thread registered. For pinned pools the window got wider in fix(cuda.core): grant device access to imported PinnedMemoryResource pools #3509, which runs the device access grant (which releases the GIL) before registration. Returning the registered resource instead of raising, as CodeRabbit suggested on fix(cuda.core): grant device access to imported PinnedMemoryResource pools #3509, removes the error but not the revoked access.

Other memory resources are not affected: VirtualMemoryResource imports (cuMemImportFromShareableHandle) give independent references and separate mappings, so closing one imported buffer leaves another import of the same descriptor usable. ManagedMemoryResource, GraphMemoryResource, and SynchronousMemoryResource have no IPC import path.

How to Reproduce

Pinned pool, which fails hard:

import multiprocessing as mp

from cuda.core import Buffer, Device, PinnedMemoryResource, PinnedMemoryResourceOptions


def child(handle, desc):
    device = Device()
    device.set_current()
    a = PinnedMemoryResource.from_allocation_handle(int(handle))
    b = PinnedMemoryResource.from_allocation_handle(int(handle))
    print("same pool handle:", int(a.handle) == int(b.handle))
    a.close()
    stream = device.default_stream
    buf = Buffer.from_ipc_descriptor(b, desc, stream=stream)  # raises CUDA_ERROR_INVALID_VALUE
    buf.fill(7, stream=stream)
    stream.sync()
    print("import through the surviving resource OK")


if __name__ == "__main__":
    mp.set_start_method("spawn")
    dev = Device()
    dev.set_current()
    mr = PinnedMemoryResource(PinnedMemoryResourceOptions(max_size=1 << 21, ipc_enabled=True))
    buf = mr.allocate(64, stream=dev.default_stream)
    dev.default_stream.sync()
    p = mp.Process(target=child, args=(mr.allocation_handle, buf.ipc_descriptor))
    p.start()
    p.join()
    buf.close()
    dev.sync()
    mr.close()

Output:

same pool handle: True
cuda.core._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_VALUE: This indicates that one or more of the parameters passed to the API call is not within an acceptable range of values.

Device pool, where a destroy fails:

import multiprocessing as mp

from cuda.core import Buffer, Device, DeviceMemoryResource, DeviceMemoryResourceOptions


def child(handle, desc):
    device = Device()
    device.set_current()
    stream = device.default_stream
    a = DeviceMemoryResource.from_allocation_handle(device, int(handle))
    b = DeviceMemoryResource.from_allocation_handle(device, int(handle))
    print("same pool handle:", int(a.handle) == int(b.handle))
    buf_a = Buffer.from_ipc_descriptor(a, desc, stream=stream)  # keeps a's pool reference alive
    a.close()
    buf = Buffer.from_ipc_descriptor(b, desc, stream=stream)
    buf.fill(7, stream=stream)
    stream.sync()
    buf.close()
    stream.sync()
    b.close()  # CUDAWarning: cuMemPoolDestroy(...) failed: CUDA_ERROR_INVALID_VALUE
    buf_a.close()


if __name__ == "__main__":
    mp.set_start_method("spawn")
    dev = Device()
    dev.set_current()
    mr = DeviceMemoryResource(dev, DeviceMemoryResourceOptions(max_size=1 << 21, ipc_enabled=True))
    buf = mr.allocate(64, stream=dev.default_stream)
    dev.default_stream.sync()
    p = mp.Process(target=child, args=(mr.allocation_handle, buf.ipc_descriptor))
    p.start()
    p.join()
    buf.close()
    dev.sync()
    mr.close()

Without buf_a, the device pool sequence completes with no error: the surviving resource keeps working and both destroys succeed.

Expected behavior

Imports of the same pool in one process share one owner, for both pinned and device pools. Device access is revoked only when the last of them is released, and no destroy runs while another import still uses the pool. A sketch:

  • Share ownership of imported pools in the C++ handle layer, keyed by CUmemoryPool: create_mempool_handle_ipc keeps a mutex-protected map from the driver handle to a weak handle, and returns the live handle when the pool is already imported. Because both resource types import through create_mempool_handle_ipc, this one change covers both.
  • The owner must still account for every import's driver reference: it counts the imports and, when its last reference is released, revokes access once and then calls cuMemPoolDestroy once per import. Destroying only once would leak a driver reference for every repeat import, and destroying a duplicate import early fails while imported buffers are outstanding.
  • An import that overlaps the final release of the same pool must wait until that release has revoked access. Otherwise the revocation can land after the new import's access grant and remove it. deviceptr_import_ipc handles the same situation for imported pointers: the entry stays in the map until its deleter has finished, and an importer that finds an expired entry waits for it.
  • With that in place, MP_from_allocation_handle can return the registered resource when a concurrent import of the same UUID registered first (keeping the class check of the registry-hit path), instead of raising. The losing import then only drops a shared reference.

Tests to add, parametrized over PinnedMemoryResource and DeviceMemoryResource:

  • A double raw-handle import followed by closing one resource, then importing a buffer through the other.
  • The same with a buffer imported through the first resource still alive when the second is closed, asserting that no CUDAWarning is emitted.
  • Concurrent imports of the same UUID from several threads, checking that the registered resource still has device access after the losing imports are dropped.

Operating System

Linux x86_64

nvidia-smi output

Driver Version: 595.99.02
GPU: NVIDIA RTX 6000 Ada Generation

cuda-core built from the #3509 branch (Nov 2026), Python 3.12.

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingcuda.coreEverything related to the cuda.core module

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions