Skip to content

[FEA]: VirtualMemoryResource unpickling: drop invisible peers and clarify the invisible-owner error #3419

Description

@juenglin

Is this a duplicate?

Area

cuda.core

Is your feature request related to a problem? Please describe.

Since #3126, a pickled VirtualMemoryResource names its owning device and its peers by UUID, and _reconstruct_virtual_memory_resource maps each UUID back through _device_from_uuid. If any of those devices is not visible to the receiving process, unpickling raises RuntimeError. This includes peers.

A Buffer pickles as (resource, descriptor), so this also affects sending a Buffer through multiprocessing. A common setup fails: a worker pinned to one GPU with CUDA_VISIBLE_DEVICES cannot receive a buffer whose sender listed other GPUs as peers, even though the worker only uses the owning device.

When the error does fire, the message comes from the generic helper:

no CUDA device visible to this process has UUID GPU-...; the visible devices are: 0 (GPU-...). Make the device visible to this process, for example through CUDA_VISIBLE_DEVICES.

It doesn't say that it failed while unpickling a VirtualMemoryResource, or whether the missing UUID is the owning device or a peer.

Describe the solution you'd like

  1. Drop peers that aren't visible instead of raising. Peer access only lets other GPUs in the receiving process use that process's mapping. A GPU the receiver can't see can't use it, so granting it access there adds nothing, and the sender's peers keep their access in the sender's process. Visibility can be checked exactly by UUID before anything is imported, so the rebuilt resource can keep only the visible peers and issue a CUDAWarning naming the ones it dropped.
  2. Raise a clearer error when the owning device isn't visible. That case still has to raise, because the receiver must map the memory with a resource for the owning device. _reconstruct_virtual_memory_resource should raise an error that says it was unpickling a VirtualMemoryResource and that the missing UUID is the device that owns the memory, instead of passing on the generic _device_from_uuid message.

The VirtualMemoryResource docstring currently says unpickling "raises RuntimeError if one of the devices is not visible" and would need updating. There are no tests for invisible peers yet; this needs tests for both cases, for example on a two-GPU runner with one GPU hidden.

Describe alternatives you've considered

  • Keep raising for invisible peers, with a clearer message. This is stricter: a missing peer may mean the processes were misconfigured. In that case the message should name the peer and point to the workaround below.
  • Current workaround: send only the descriptor, not the Buffer, and import it with Buffer.from_ipc_descriptor() using a VirtualMemoryResource built in the receiving process for the owning device, with peers that exist there. This works today, but a POSIX file-descriptor descriptor carries no owner UUID, so the receiver must learn the owning device some other way.

Additional context

Follow-up to #3028 and #3126, discussed with @Andy-Jost.

Activity

  1. added
    enhancementAny code-related improvements
    cuda.coreEverything related to the cuda.core module
    on Nov 10, 2026
  2. Andy-Jost commented on Nov 16, 2026

    @Andy-Jost
    Contributor

    Accepting this for cuda.core next. Both parts are right: a peer the receiver cannot see cannot use the receiver's mapping, so dropping it loses nothing, and the invisible-owner case must keep raising. Three constraints for the PR. First, keep RuntimeError for the invisible owner, because the docstring and the 1.3.0 notes promise that type; only the message changes, and it should use the wording of the from_ipc_descriptor error ("memory belongs to the device with UUID ..., which is not visible in this process"). Second, issue one CUDAWarning that names the dropped peer UUIDs, and document that the rebuilt resource lists only the visible peers, because pickling it again forwards the shorter list. Third, tests: tests/memory_ipc/test_device_uuid.py already has the hidden-device harness for the two-GPU cases, and a single-GPU test can call _reconstruct_virtual_memory_resource with a made-up peer UUID, so most of the coverage needs no two-GPU runner. Update the class docstring and add a release note.

  3. added this to the cuda.core next milestone on Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P2Low priority - Nice to havecuda.coreEverything related to the cuda.core moduleenhancementAny code-related improvements

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions