Skip to content

fix(cuda.core): reject importing a memory pool as the other resource class - #3516

Merged
juenglin merged 6 commits into
NVIDIA:mainfrom
juenglin:fix-ipc-import-pool-location
Nov 21, 2026
Merged

juenglin merged 6 commits into
NVIDIA:mainfrom
juenglin:fix-ipc-import-pool-location

Conversation

@juenglin

@juenglin juenglin commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Description

closes #3502

DeviceMemoryResource.from_allocation_handle accepted the allocation handle of a host (pinned) pool, and PinnedMemoryResource.from_allocation_handle the handle of a device pool, unless the resource was already registered in the importing process. The result misreported device_id, is_host_accessible and is_device_accessible, and for a device pool imported as pinned, the device access grant from #3509 ran on a pool it was not written for.

Fix

  • Both imports check the pool's CU_MEMPOOL_ATTR_LOCATION_TYPE in the import hook, before registration and before the pinned access grant, so raw file descriptors and unregistered handles are covered. A mismatch raises TypeError. Because repeat imports share one owner (fix(cuda.core): share ownership of repeat imports of an IPC memory pool #3511), a rejected import only drops a reference and leaves an earlier import of the same pool intact.
  • Accepted locations: HOST and HOST_NUMA for PinnedMemoryResource; DEVICE, INVISIBLE and DEVICE_LOCALITY_DOMAIN for DeviceMemoryResource. Anything else raises TypeError.
  • An imported PinnedMemoryResource takes numa_id from CU_MEMPOOL_ATTR_LOCATION_ID instead of reporting 0, and grants access for the pool's actual location type.
  • The check and the numa_id lookup need a CUDA 13 build and a 13.2+ driver; otherwise imports behave as before.

Decisions

The pool's device is not checked against device_id, and a pinned import with no supporting visible device keeps host-only access; see the comment on #3502.

Tests

New cuda_core/tests/memory_ipc/test_ipc_pool_kind.py. The first three tests skip on a CUDA 12 build or a driver before 13.2; the last two run everywhere:

  • A mismatched import raises TypeError in both directions, leaves no pool behind and emits no CUDAWarning.
  • A rejected import leaves an earlier import of the same pool usable: every device keeps its access, including peer access, and a buffer imported before the rejection still works.
  • An imported pinned resource reports a nonzero exporter numa_id; skips without a second NUMA node.
  • A matching-class import works.
  • With the driver version forced below 13.2 (no CI row runs a pre-13.2 driver), the pool's kind is not checked and imports behave as before.

Not tested on a multi-domain GPU; the peer-access check needs more than one GPU.

Checklist

  • I have read the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@juenglin juenglin added this to the cuda.core 1.4.0 milestone Nov 20, 2026
@juenglin juenglin added bug Something isn't working cuda.core Everything related to the cuda.core module labels Nov 20, 2026
@copy-pr-bot

copy-pr-bot Bot commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@juenglin juenglin self-assigned this Nov 20, 2026
juenglin added a commit to juenglin/cuda-python that referenced this pull request Nov 20, 2026
…nd its location types

Address review of NVIDIA#3516.

- CU_MEMPOOL_ATTR_LOCATION_TYPE and _ID are 13.2 driver attributes. A 13.0 or
  13.1 driver rejects them, which would have failed every pool import. Gate the
  check and the numa_id lookup on cy_driver_version() >= (13, 2, 0), like other
  13.2 APIs, and fall back to the CUDA 12 behavior otherwise.
- Name the accepted location types: HOST and HOST_NUMA for a pinned import,
  DEVICE and INVISIBLE (a device that is not visible to the importer) for a
  device import. Any other location raises TypeError.
- Skip the new tests on the driver version, and add a test that importing with
  the exporter's class works whether or not the check is available.
- Say "CUDA 13 build and 13.2 driver" in the docstrings and release note, and
  document the numa_id fallback.
@juenglin

Copy link
Copy Markdown
Contributor Author

/ok to test 2d690fe

@juenglin
juenglin requested review from Andy-Jost and rwgk November 20, 2026 03:22
@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor
Doc Preview CI
Preview removed because the pull request was closed or merged.

@rwgk

rwgk commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

codex GPT-6.2-Sol ultra:

  • P2. Restack on fix(cuda.core): share ownership of repeat imports of an IPC memory pool #3511’s registry fix. Free-threaded CI on CUDA 12 and 13 shows concurrent imports of the same UUID returning different objects for both resource classes. The reviewed revision predates fix(cuda.core): share ownership of repeat imports of an IPC memory pool #3511’s fix, which locks the registry lookup and insertion together. CUDA 12 CI · CUDA 13 CI · Base fix
  • P2. Avoid the unsynchronized driver-version cache in concurrent imports. The new location check calls cy_driver_version(). On free-threaded Python, two first-time importers can race while replacing the cached tuple, corrupting its reference count. Use the existing functools.cache-backed driver_version() or synchronize the Cython cache. New call site · Cache implementation
  • P2. Accept device locality-domain pools. If an imported pool reports CU_MEM_LOCATION_TYPE_DEVICE_LOCALITY_DOMAIN, the new check raises TypeError even when the caller correctly chooses DeviceMemoryResource. CUDA documents that single-domain GPUs report DEVICE for these pools, so this needs verification on a multi-domain GPU. Location check · CUDA Driver API
  • P3. Exercise a nonzero NUMA ID. The regression test passes with the old always-zero behavior whenever the exporter uses node 0. Select a supported nonzero node and skip when none is available. NUMA test

Reviewed PR #3516 at 2d690fe. Cython generation and C++ syntax checks passed against CUDA 12.9 and 13.5. All seven new pool-kind cases passed in the CUDA 13 A100 job.

juenglin added a commit to juenglin/cuda-python that referenced this pull request Nov 20, 2026
…nd its location types

Address review of NVIDIA#3516.

- CU_MEMPOOL_ATTR_LOCATION_TYPE and _ID are 13.2 driver attributes. A 13.0 or
  13.1 driver rejects them, which would have failed every pool import. Gate the
  check and the numa_id lookup on cy_driver_version() >= (13, 2, 0), like other
  13.2 APIs, and fall back to the CUDA 12 behavior otherwise.
- Name the accepted location types: HOST and HOST_NUMA for a pinned import,
  DEVICE and INVISIBLE (a device that is not visible to the importer) for a
  device import. Any other location raises TypeError.
- Skip the new tests on the driver version, and add a test that importing with
  the exporter's class works whether or not the check is available.
- Say "CUDA 13 build and 13.2 driver" in the docstrings and release note, and
  document the numa_id fallback.
juenglin added a commit to juenglin/cuda-python that referenced this pull request Nov 20, 2026
…ion safely in the pool kind check

Address review of NVIDIA#3516:

- DeviceMemoryResource now accepts an imported pool at
  CU_MEM_LOCATION_TYPE_DEVICE_LOCALITY_DOMAIN, which lives in part of a
  device's memory.
- MP_location_query_available reads the functools-cached driver_version()
  instead of cy_driver_version(), whose cache is filled without
  synchronization. Concurrent first imports on free-threaded builds could
  otherwise race on it.
- The imported numa_id test exports from a nonzero NUMA node with memory,
  because the old behavior reported 0, and skips when there is none.
juenglin added a commit to juenglin/cuda-python that referenced this pull request Nov 20, 2026
…until NVIDIA#3510 is fixed

Retarget NVIDIA#3516 from NVIDIA#3511 to main. Without NVIDIA#3511, a second import of a pool
in one process gets its own owner of the shared driver pool (NVIDIA#3510).
Dropping a rejected import of the pinned pool then releases that pool and
revokes the device access of the earlier import. Mark that case as a strict
xfail; the XPASS on top of NVIDIA#3511 flags the marker for removal there.

Also drop a doubled blank line in the 1.4.0 release notes left by the rebase.
@juenglin
juenglin force-pushed the fix-ipc-import-pool-location branch from 2d690fe to 721d0d9 Compare November 20, 2026 22:18
@juenglin
juenglin changed the base branch from pull-request/3511 to main November 20, 2026 22:19
@github-actions github-actions Bot added the CI/CD CI/CD infrastructure label Nov 20, 2026
@juenglin

Copy link
Copy Markdown
Contributor Author

Thanks! Addressed in 74c5167 and 721d0d9. The PR now targets main instead of #3511.

@coderabbitai

coderabbitai Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Summary

Summary by CodeRabbit

  • Bug Fixes

    • Importing a host memory-pool handle as a device resource, or a device-pool handle as a pinned resource, now raises TypeError when pool-kind checking is available.
    • Imported pinned resources report the pool’s NUMA node when location queries are supported; otherwise, the reported node may be 0.
  • Documentation

    • Clarified supported import types, NUMA reporting, and platform requirements. Pool-kind checks and NUMA lookup require a CUDA 13 build and driver 13.2 or later; otherwise, import behavior remains unchanged.

Memory-pool imports now check whether the pool location matches the requested resource class when location queries are supported. Pinned-resource imports also use available pool location data to set the reported NUMA ID and device-access capabilities. Tests and release notes cover these changes and their CUDA version requirements.

Changes: Memory pool imports

Layer / File(s) Summary
Location query support
cuda_core/cuda/core/_memory/_memory_pool.pxd, cuda_core/cuda/core/_memory/_memory_pool.pyx
Adds helpers that check whether location queries are supported and retrieve a pool’s location. Queries require a CUDA 13 build and driver 13.2 or later.
Import validation and pinned-resource initialization
cuda_core/cuda/core/_memory/_ipc.pxd, cuda_core/cuda/core/_memory/_ipc.pyx, cuda_core/cuda/core/_memory/_device_memory_resource.pyx, cuda_core/cuda/core/_memory/_device_memory_resource.pyi, cuda_core/cuda/core/_memory/_pinned_memory_resource.pyx, cuda_core/cuda/core/_memory/_pinned_memory_resource.pyi
Import callbacks reject a pool whose location does not match the requested resource class when queries are available. Pinned-resource imports use queried location data to set the NUMA ID and device-access capabilities. The API documentation describes these behaviors and their limits.
Tests and release notes
cuda_core/tests/helpers/memory.py, cuda_core/tests/memory_ipc/test_ipc_pool_kind.py, cuda_core/docs/source/release/1.4.0-notes.rst
Adds coverage for matching and mismatched imports, existing imports, and imported pinned-resource NUMA IDs. Documents the behavior and CUDA version requirements.

Assessment against linked issues

Objective Addressed Explanation
Reject imports when the pool kind does not match the requested resource class [#3502] ✅

Out-of-scope changes

Code Change Explanation
Set imported pinned-resource NUMA ID and device-access capabilities from the pool location (cuda_core/cuda/core/_memory/_pinned_memory_resource.pyx:369-378) The linked issue focuses on rejecting pool-kind mismatches and identifies imported pinned-resource access as a separate defect.

Possibly related issues

  • NVIDIA/cuda-python#3508: The change sets an imported pinned resource’s numa_id from the pool location when location queries are supported.

Change: Bug fix · Severity of issue fixed: Medium

Merge Risk: Low · up to e32cc

This change makes importing a memory pool as the wrong resource class raise TypeError on CUDA 13 builds with a 13.2+ driver, and it reports the pool's NUMA ID for pinned imports. One known limitation remains. A rejected repeat import of an already-imported pool can still invalidate the earlier import until #3511 lands. The author documents this and covers it with a strict xfail, so the merge risk is low.

Security architecture risk: Minimal · up to e32cc

The change strengthens pool-type validation before registration and pinned-memory access grants. Plain-host imports use the existing process-local access policy for compatible devices. No introduced or worsened ownership or security-boundary failure was established in the inspected changes.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The affected authority is access to the imported pool within the importing process, including compatible visible devices. The inspected path requires a shareable allocation handle accepted by the CUDA driver; it does not establish access to unrelated pools, tenants, or remote services.

Trust Boundaries and Controls

  • observed — The new classifier uses driver-reported pool location rather than trusting the caller's chosen resource class. On supported environments, importing a device pool as pinned raises before the pinned read/write grant can execute.

Resilience and Maintainability Implications

  • observed — New tests encode rejected-import cleanup and preservation of an existing import's access flags and live buffer usability. They support the intended ownership invariant, but were inspected rather than executed.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 721d0d9 and e32cc25.

Files selected for processing: 9

  • cuda_core/cuda/core/_memory/_device_memory_resource.pyi
  • cuda_core/cuda/core/_memory/_device_memory_resource.pyx
  • cuda_core/cuda/core/_memory/_ipc.pxd
  • cuda_core/cuda/core/_memory/_ipc.pyx
  • cuda_core/cuda/core/_memory/_memory_pool.pxd
  • cuda_core/cuda/core/_memory/_memory_pool.pyx
  • cuda_core/cuda/core/_memory/_pinned_memory_resource.pyx
  • cuda_core/docs/source/release/1.4.0-notes.rst
  • cuda_core/tests/memory_ipc/test_ipc_pool_kind.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between de0af8d and 721d0d9.

Files selected for processing: 11

  • cuda_core/cuda/core/_memory/_device_memory_resource.pyi
  • cuda_core/cuda/core/_memory/_device_memory_resource.pyx
  • cuda_core/cuda/core/_memory/_ipc.pxd
  • cuda_core/cuda/core/_memory/_ipc.pyx
  • cuda_core/cuda/core/_memory/_memory_pool.pxd
  • cuda_core/cuda/core/_memory/_memory_pool.pyx
  • cuda_core/cuda/core/_memory/_pinned_memory_resource.pyi
  • cuda_core/cuda/core/_memory/_pinned_memory_resource.pyx
  • cuda_core/docs/source/release/1.4.0-notes.rst
  • cuda_core/tests/helpers/memory.py
  • cuda_core/tests/memory_ipc/test_ipc_pool_kind.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@juenglin

Copy link
Copy Markdown
Contributor Author

/ok to test 721d0d9

@rwgk rwgk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

codex GPT-6.2-Sol ultra:

Reviewed PR #3516 at HEAD 721d0d9.

  • [P2] Land #3511 before enabling wrong-class rejection. The cleanup problem also affects device pools. If an existing import grants access to a peer GPU, retrying the same raw FD as PinnedMemoryResource raises TypeError and releases a second owner of the same driver pool. That release revokes the existing import’s peer access. With a live imported buffer, cuMemPoolDestroy can also fail and leave the extra driver reference behind, even on a single GPU. The current test checks the resident GPU and imports its buffer after the rejection, so it misses both cases. Import and validation · Regression test
  • [P3] Assert that rejected imports emit no CUDAWarning. The live-pool counter decrements even if cuMemPoolDestroy fails, and the spawned child can emit the warning and exit successfully. Wrap the rejected calls in assert_no_cuda_warning() inside the child so the test catches a failed rollback. Cleanup assertion · Pool deleter

@juenglin
juenglin marked this pull request as draft November 21, 2026 01:27
juenglin added a commit to juenglin/cuda-python that referenced this pull request Nov 21, 2026
…ol (NVIDIA#3511, squashed)

Temporary squash of NVIDIA#3511 at dc522f7 onto main, standing in for its
merge so that NVIDIA#3516 can be prepared on top. Not for pushing.
juenglin added a commit to juenglin/cuda-python that referenced this pull request Nov 21, 2026
…nd its location types

Address review of NVIDIA#3516.

- CU_MEMPOOL_ATTR_LOCATION_TYPE and _ID are 13.2 driver attributes. A 13.0 or
  13.1 driver rejects them, which would have failed every pool import. Gate the
  check and the numa_id lookup on cy_driver_version() >= (13, 2, 0), like other
  13.2 APIs, and fall back to the CUDA 12 behavior otherwise.
- Name the accepted location types: HOST and HOST_NUMA for a pinned import,
  DEVICE and INVISIBLE (a device that is not visible to the importer) for a
  device import. Any other location raises TypeError.
- Skip the new tests on the driver version, and add a test that importing with
  the exporter's class works whether or not the check is available.
- Say "CUDA 13 build and 13.2 driver" in the docstrings and release note, and
  document the numa_id fallback.
juenglin added a commit to juenglin/cuda-python that referenced this pull request Nov 21, 2026
…ion safely in the pool kind check

Address review of NVIDIA#3516:

- DeviceMemoryResource now accepts an imported pool at
  CU_MEM_LOCATION_TYPE_DEVICE_LOCALITY_DOMAIN, which lives in part of a
  device's memory.
- MP_location_query_available reads the functools-cached driver_version()
  instead of cy_driver_version(), whose cache is filled without
  synchronization. Concurrent first imports on free-threaded builds could
  otherwise race on it.
- The imported numa_id test exports from a nonzero NUMA node with memory,
  because the old behavior reported 0, and skips when there is none.
@juenglin
juenglin force-pushed the fix-ipc-import-pool-location branch from 721d0d9 to 7594ceb Compare November 21, 2026 01:52
juenglin added a commit to juenglin/cuda-python that referenced this pull request Nov 21, 2026
… a rejected pool import

Address review of NVIDIA#3516: the rejection test now keeps an imported buffer
alive across the rejected import, grants peer access on a device pool when a
peer can use it, and compares every device's pool access before and after.
Rejected imports must also emit no CUDAWarning, since a failed
cuMemPoolDestroy during their release surfaces only as a warning.
@juenglin

Copy link
Copy Markdown
Contributor Author

/ok to test 0784de8

…class (NVIDIA#3502)

DeviceMemoryResource.from_allocation_handle accepted the handle of a host
pool, and PinnedMemoryResource.from_allocation_handle the handle of a device
pool, unless the resource was already registered in the importing process.
The result misreported device_id and host/device accessibility.

Both imports now check CU_MEMPOOL_ATTR_LOCATION_TYPE in their init hook,
before the resource is registered and before the pinned access grant runs,
and raise TypeError on a mismatch. An imported PinnedMemoryResource takes
numa_id from CU_MEMPOOL_ATTR_LOCATION_ID and grants access for the pool's
actual location type instead of assuming a host NUMA pool. The attributes
exist only in the CUDA 13 driver API; a CUDA 12 build keeps the previous
behavior.
…nd its location types

Address review of NVIDIA#3516.

- CU_MEMPOOL_ATTR_LOCATION_TYPE and _ID are 13.2 driver attributes. A 13.0 or
  13.1 driver rejects them, which would have failed every pool import. Gate the
  check and the numa_id lookup on cy_driver_version() >= (13, 2, 0), like other
  13.2 APIs, and fall back to the CUDA 12 behavior otherwise.
- Name the accepted location types: HOST and HOST_NUMA for a pinned import,
  DEVICE and INVISIBLE (a device that is not visible to the importer) for a
  device import. Any other location raises TypeError.
- Skip the new tests on the driver version, and add a test that importing with
  the exporter's class works whether or not the check is available.
- Say "CUDA 13 build and 13.2 driver" in the docstrings and release note, and
  document the numa_id fallback.
…ion safely in the pool kind check

Address review of NVIDIA#3516:

- DeviceMemoryResource now accepts an imported pool at
  CU_MEM_LOCATION_TYPE_DEVICE_LOCALITY_DOMAIN, which lives in part of a
  device's memory.
- MP_location_query_available reads the functools-cached driver_version()
  instead of cy_driver_version(), whose cache is filled without
  synchronization. Concurrent first imports on free-threaded builds could
  otherwise race on it.
- The imported numa_id test exports from a nonzero NUMA node with memory,
  because the old behavior reported 0, and skips when there is none.
… a rejected pool import

Address review of NVIDIA#3516: the rejection test now keeps an imported buffer
alive across the rejected import, grants peer access on a device pool when a
peer can use it, and compares every device's pool access before and after.
Rejected imports must also emit no CUDAWarning, since a failed
cuMemPoolDestroy during their release surfaces only as a warning.
…before 13.2

Address review of NVIDIA#3516: no CI row runs the cuda.core tests with a CUDA 13
build on a driver older than 13.2, so the driver_version() < (13, 2, 0)
branch of MP_location_query_available never ran. A spawned child now
replaces the driver_version that _memory_pool reads, as
test_program_compile_cache.py does for _program, and checks the documented
fallback: the other class's import is accepted without a CUDAWarning, the
matching import works, and an imported pinned resource reports numa_id 0.
@juenglin
juenglin force-pushed the fix-ipc-import-pool-location branch from 0784de8 to e32cc25 Compare November 21, 2026 03:16
@juenglin

Copy link
Copy Markdown
Contributor Author

/ok to test e32cc25

@juenglin
juenglin marked this pull request as ready for review November 21, 2026 03:28
@juenglin
juenglin enabled auto-merge (squash) November 21, 2026 03:32
@juenglin
juenglin merged commit edf1b31 into NVIDIA:main Nov 21, 2026
125 checks passed
github-actions Bot pushed a commit that referenced this pull request Nov 21, 2026
Removed preview folders for the following PRs:
- PR #2924
- PR #3470
- PR #3490
- PR #3511
- PR #3515
- PR #3516
- PR #3520
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working CI/CD CI/CD infrastructure cuda.core Everything related to the cuda.core module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: from_allocation_handle silently imports a memory pool as the wrong resource class

2 participants