Skip to content

fix(cuda.core): pick one owner when IPC pool registrations of one UUID race - #3524

Open
juenglin wants to merge 2 commits into
NVIDIA:mainfrom
juenglin:fix-ipc-registry-reentry-and-gpu-check
Open

juenglin wants to merge 2 commits into
NVIDIA:mainfrom
juenglin:fix-ipc-registry-reentry-and-gpu-check

Conversation

@juenglin

@juenglin juenglin commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Description

Follow-up to #3511, which closed #3510 and deferred two review items so it could merge without another CI round while #3516 waited on it. This PR addresses both.

Pick one owner when registrations of one UUID race in MP_register. Repeat imports of one IPC pool share a single owner keyed by the pool's UUID. Selecting that owner was not atomic against same-thread reentry: inserting into the registry runs Python code (the WeakValueDictionary methods and uuid.UUID.__hash__), where a garbage collection can start and its callbacks can register the same UUID on the same thread (the lock is reentrant). The re-entrant call inserted its resource, then the outer call overwrote it, leaving two owners of one driver pool with the registry holding only one of them. The overwrite predates #3511; main registered with no lock at all. MP_register now records the resource it is inserting, under _registry_lock, before it inserts. A re-entrant registration of the same UUID returns that resource instead of inserting its own, and one that completed before the record was set is found by a second lookup. Both calls then return the same owner, and the registry holds it.

Check every supported GPU after the re-import race. test_reimport_racing_the_last_release_keeps_access checked read/write access and exercised a buffer only on the default GPU. The final release of a pinned pool revokes access one GPU at a time, so a re-import could in principle be left without access on a non-default GPU and the test would still pass. It now asserts read/write access on every GPU the pool grants access to.

Notes for reviewers

Please review the combined diff. The first commit replaced the registry with a plain dict of weak references whose removal callback took _registry_lock. That made every drop of a registered resource wait for registrations in progress on other threads, and a thread that dropped one while holding an application lock deadlocked with a registration whose garbage-collection finalizer took the same lock. The second commit restores the WeakValueDictionary and fixes the overwrite inside MP_register instead.

Testing

  • New test_reentrant_registration_of_one_uuid_picks_one_winner drives a deterministic regression for the overwrite through a one-shot, test-only hook in MP_register (_set_register_reentry_hook_for_testing) that registers the same UUID re-entrantly right before the outer call's insertion, where a finalizer could. With the code on main plus the hook, it fails on assert outer is winner for both PinnedMemoryResource and DeviceMemoryResource; with the fix it passes.
  • tests/memory_ipc passes locally on driver 595.99.02 (single GPU): 171 passed, 46 skipped.

Scope

cuda.core only. The registry and MP_register have no counterpart in cuda_bindings_12/ or cuda_bindings_13/, and the touched files contain no BEGIN SYNCED groups. No release-note entry is added: the winner-selection fix hardens a latent, essentially unobservable condition already covered in spirit by the #3510 note on concurrent imports sharing one resource, and the GPU check is test-only.

Checklist

  • I have read the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

…D race

Follow-up to NVIDIA#3511 (NVIDIA#3510). Two deferred review items:

- MP_register could overwrite a re-entrant registration of the same UUID.
  A garbage-collection finalizer that registered the same pool between the
  outer call's lookup and its insert left two owners of one driver pool, with
  the registry pointing at the loser. Build the weak reference before the
  insert and claim the slot with dict.setdefault so a re-entrant winner is
  kept, not overwritten; both calls then return that one owner. A plain dict
  of weak references replaces the WeakValueDictionary so the wrapper
  allocation happens outside the check-and-set. A one-shot test hook injects
  the reentry at the vulnerable point for a deterministic regression test.

- The re-import race test now checks read/write access on every GPU the pool
  grants access to, not only the default one, since the final release revokes
  access one GPU at a time.
@juenglin juenglin added this to the cuda.core 1.4.0 milestone Nov 21, 2026
@juenglin juenglin added bug Something isn't working cuda.core Everything related to the cuda.core module labels Nov 21, 2026
@juenglin juenglin self-assigned this Nov 21, 2026
@copy-pr-bot

copy-pr-bot Bot commented Nov 21, 2026

Copy link
Copy Markdown
Contributor

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Explain Change →

Summary

Summary by CodeRabbit

  • Bug Fixes
    • Improved reliability when importing shared memory pools, including cases where multiple registrations occur concurrently or are nested. Imports now consistently use the same registered pool, helping prevent conflicts and resource leaks.
    • Expanded memory-pool access checks to cover every device supported by a pool, improving confidence that access remains available across supported devices.

The IPC registry now tracks UUIDs with registrations in progress and handles re-entrant registration without replacing the resource already registered. Tests cover the nested registration case and check pool access on supported devices.

Changes: IPC registry registration

Layer / File(s) Summary
Track in-progress registrations
cuda_core/cuda/core/_memory/_ipc.pyx, cuda_core/cuda/core/_memory/_ipc.pyi
MP_register checks for in-progress registrations, marks the UUID before insertion, and clears the marker after registration. The stub declares the test hook used before insertion.
Registration and pool access tests
cuda_core/tests/helpers/memory.py, cuda_core/tests/memory_ipc/test_ipc_pool_duplicate_import.py
A helper selects devices supported by each pool type. Tests trigger nested registration for the same UUID and verify that both calls return the same resource. The access test checks all supported devices.

Linked issue assessment

Objective Addressed Explanation
Concurrent same-UUID registration returns the registered resource rather than failing. [#3510] ✅
The registered resource retains device access after a competing import is dropped. [#3510] ❓ The tests check the registration winner and owned-pool count, but do not check access after dropping a competing import.
Repeated raw-handle imports share ownership and avoid premature pool destruction with outstanding buffers. [#3510] ❌ The diff does not change raw-handle imports or native pool ownership and destruction.

Possibly related PRs

  • NVIDIA/cuda-python#3511: It changes the same IPC registration path and ownership contract. This PR follows up on re-entrant registration behavior.

Suggested reviewers: andy-jost, mdboom

Change: Bug fix · Severity of issue fixed: Medium

Merge Risk: Low · up to e04e2

This change makes nested registrations of the same IPC pool UUID return a single owner. A narrow window remains where the nested caller receives that owner before it is fully registered. The impact is limited, so the change is mergeable with owner awareness.

Security architecture risk: Low · up to e04e2

Normal reentrant registrations now converge on one resource, but a nested caller receives that resource before its registration and identity are committed. This leaves a process-local ownership consistency gap. No expansion of external access or privileges was identified.

Retained concerns

  • Low · reliability · inferred: The in-progress branch returns the outer owner before committing its UUID or registry entry. For a raw-descriptor import whose UUID is initially None, a reentrant caller can register that returned owner under another UUID; the outer call then assigns its original UUID, leaving two registry keys for one object with a changed identity. If the outer insertion raises instead, the nested caller can retain an owner that was never registered. This weakens identity and recovery guarantees used by UUID-based serialization and reconstruction. At the base, the corresponding nested registration completed its own identity assignment before returning.
Security review details

Security Blast Radius

  • inferred — The changed ownership state is process-local and UUID-keyed. Affected resources can represent device pools or pinned pools; pinned-pool imports already grant read/write access to every supporting GPU in the process. The identified consistency concern does not establish additional GPU permissions or access to another process's pool.

Trust Boundaries and Controls

  • inferred — The new in-progress return requires same-thread reentry while registration holds its lock, such as an executing finalizer or the local testing hook. The inspected change does not introduce a network entrypoint or a mechanism by which an IPC peer alone can install that callback.

Resilience and Maintainability Implications

  • inferred — The unchanged C++ import layer reuses shared ownership for repeated imports of the same driver pool. Its final shared-owner deleter revokes access and destroys the counted imports; closing a Python resource releases its handle reference. This limits the identified Python registry inconsistency and is counterevidence against inferring double destruction or premature access revocation from duplicate Python owners alone.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

The previous commit replaced the registry with a plain dict of weak
references whose removal callback took _registry_lock. That callback runs
wherever the last reference to a registered resource is dropped, including
in garbage collection on any thread, so dropping one now waited for any
registration in progress on another thread. A thread holding an application
lock while dropping a registered resource deadlocked with a registration
whose garbage-collection finalizer took the same lock. WeakValueDictionary
removes entries without a lock, through an atomic helper, for exactly this
reason.

Restore the WeakValueDictionary and its lookups unchanged. MP_register now
marks the resource it is inserting, under _registry_lock, before it inserts:
a registration of the same UUID re-entered from a garbage-collection
callback returns the marked resource instead of inserting its own, and one
that completed before the mark is found by a second lookup.
@juenglin

Copy link
Copy Markdown
Contributor Author

/ok to test e04e278

@juenglin
juenglin enabled auto-merge (squash) November 21, 2026 04:08
@juenglin
juenglin requested review from Andy-Jost and rwgk November 21, 2026 04:08

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 3275ccf and e04e278.

Files selected for processing: 3

  • cuda_core/cuda/core/_memory/_ipc.pyi
  • cuda_core/cuda/core/_memory/_ipc.pyx
  • cuda_core/tests/memory_ipc/test_ipc_pool_duplicate_import.py
Files with no reviewable changes: 1

  • cuda_core/cuda/core/_memory/_ipc.pyi
Files skipped from review as they are similar to previous changes: 1

  • cuda_core/tests/memory_ipc/test_ipc_pool_duplicate_import.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 8 remain after this review.

if existing is None:
# A garbage-collection callback re-entered MP_register while this
# thread is inserting another resource under this UUID.
existing = _registering.get(uuid)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor · Data Integrity & Integration · Quick win

important: Complete registration before returning the marked owner.

With the supplied raw-fd imports, nested second.register(U) returns first before Line 665 inserts it and Line 671 sets its UUID. Inside that callback, mr_type.from_registry(U) therefore raises, although register() promises a registered resource. first.uuid also remains None until the outer call resumes. (raw.githubusercontent.com)

Make the registry entry and UUID usable before the nested call returns. Add both assertions inside the re-entry callback; assertions after the outer call cannot detect this failure.

@github-actions

Copy link
Copy Markdown
Contributor

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working cuda.core Everything related to the cuda.core module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: Importing a memory pool twice in one process creates two owners of one driver pool

1 participant