Skip to content

test(cuda.core): cover re-importing a pinned pool after a release with a pending free - #3531

Draft
juenglin wants to merge 3 commits into
NVIDIA:mainfrom
juenglin:test-3514-reimport-pending-free
Draft

juenglin wants to merge 3 commits into
NVIDIA:mainfrom
juenglin:test-3514-reimport-pending-free

Conversation

@juenglin

@juenglin juenglin commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Description

closes #3514

Adds a regression test for re-importing a pinned pool after a release with an unsynchronized free still pending. The fix already landed in #3511 (the CUDA_ERROR_ALREADY_MAPPED retry in MP_set_access). Only the existing race test catches its removal, and only as a side effect of its timing scenario, so this tests the sequence from the issue directly.

The existing child functions in the module now release their resources through an ExitStack when a test fails.

The test runs in a child process and is parametrized over the import paths named in the issue: a single import, several imports of one raw handle, and concurrent imports of one UUID. It asserts that the re-import grants read/write access, that a buffer imported through it can be filled, and that no CUDAWarning is issued.

Checklist

  • I have read the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

…h a pending free

Releasing the last import of a pool while an unsynchronized
cuMemFreeAsync is still pending leaves the driver holding the pool. The
next import returns the same CUmemoryPool, and its first cuMemPoolSetAccess
fails with CUDA_ERROR_ALREADY_MAPPED. The retry added in NVIDIA#3511 handles
this, but no test exercised the sequence.

The new test imports a pool, fills a buffer, closes the buffer without
synchronizing, releases every resource, and imports the pool again. It
asserts that the re-import grants read/write access, that a buffer
imported through it can be filled, and that no CUDAWarning is issued. It
covers a single import, several imports of one raw handle, and concurrent
imports of one UUID.

Without the retry, the three PinnedMemoryResource cases fail with
CUDA_ERROR_ALREADY_MAPPED. The DeviceMemoryResource cases pass either way,
since a device pool import grants no access.

Closes NVIDIA#3514.

Signed-off-by: Ralf Juengling <rjuengling@nvidia.com>
@juenglin juenglin added this to the cuda.core 1.4.0 milestone Nov 21, 2026
@juenglin juenglin added test Improvements or additions to tests cuda.core Everything related to the cuda.core module labels Nov 21, 2026
@juenglin juenglin self-assigned this Nov 21, 2026
@copy-pr-bot

copy-pr-bot Bot commented Nov 21, 2026

Copy link
Copy Markdown
Contributor

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

The child functions in test_ipc_pool_duplicate_import.py closed their
resources and buffers at the end of the body, so a failed assertion or a
failed second import left them open. They now enter each one into an
ExitStack as it is created: closing(resource) for the pool resources,
which are not context managers, and the buffer itself. The stack closes in
reverse order, so buffers still close before the resources they came
from. Closes that are the sequence under test stay explicit, and the stack
repeats them as no-ops.

_import_twice takes the stack, so the first import is released if the
second fails. _import_concurrently closes the imports that succeeded
before it raises. The race test also owns the buffer, resource and stream
that its re-import thread creates, and still synchronizes the stream
before closing it.

Signed-off-by: Ralf Juengling <rjuengling@nvidia.com>
Three helpers remove repetition across the child functions: _enter_import wraps an import in contextlib.closing on an ExitStack, _run_threads starts and joins a list of threads, and the _child_device context manager sets the current device and asserts the owned-pool count returns to its baseline on exit. The child functions use them instead of repeating the setup, the thread start/join loops, and the baseline bookkeeping.

Signed-off-by: Ralf Juengling <rjuengling@nvidia.com>
@juenglin

Copy link
Copy Markdown
Contributor Author

/ok to test 56274e3

@github-actions

Copy link
Copy Markdown
Contributor

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cuda.core Everything related to the cuda.core module test Improvements or additions to tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: Re-importing a PinnedMemoryResource pool fails with CUDA_ERROR_ALREADY_MAPPED after a release with a pending buffer free

1 participant