Repository navigation
Conversation
…h a pending free Releasing the last import of a pool while an unsynchronized cuMemFreeAsync is still pending leaves the driver holding the pool. The next import returns the same CUmemoryPool, and its first cuMemPoolSetAccess fails with CUDA_ERROR_ALREADY_MAPPED. The retry added in NVIDIA#3511 handles this, but no test exercised the sequence. The new test imports a pool, fills a buffer, closes the buffer without synchronizing, releases every resource, and imports the pool again. It asserts that the re-import grants read/write access, that a buffer imported through it can be filled, and that no CUDAWarning is issued. It covers a single import, several imports of one raw handle, and concurrent imports of one UUID. Without the retry, the three PinnedMemoryResource cases fail with CUDA_ERROR_ALREADY_MAPPED. The DeviceMemoryResource cases pass either way, since a device pool import grants no access. Closes NVIDIA#3514. Signed-off-by: Ralf Juengling <rjuengling@nvidia.com>
Contributor
The child functions in test_ipc_pool_duplicate_import.py closed their resources and buffers at the end of the body, so a failed assertion or a failed second import left them open. They now enter each one into an ExitStack as it is created: closing(resource) for the pool resources, which are not context managers, and the buffer itself. The stack closes in reverse order, so buffers still close before the resources they came from. Closes that are the sequence under test stay explicit, and the stack repeats them as no-ops. _import_twice takes the stack, so the first import is released if the second fails. _import_concurrently closes the imports that succeeded before it raises. The race test also owns the buffer, resource and stream that its re-import thread creates, and still synchronizes the stream before closing it. Signed-off-by: Ralf Juengling <rjuengling@nvidia.com>
Three helpers remove repetition across the child functions: _enter_import wraps an import in contextlib.closing on an ExitStack, _run_threads starts and joins a list of threads, and the _child_device context manager sets the current device and asserts the owned-pool count returns to its baseline on exit. The child functions use them instead of repeating the setup, the thread start/join loops, and the baseline bookkeeping. Signed-off-by: Ralf Juengling <rjuengling@nvidia.com>
Contributor
Author
|
/ok to test 56274e3 |
Contributor
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
closes #3514
Adds a regression test for re-importing a pinned pool after a release with an unsynchronized free still pending. The fix already landed in #3511 (the
CUDA_ERROR_ALREADY_MAPPEDretry inMP_set_access). Only the existing race test catches its removal, and only as a side effect of its timing scenario, so this tests the sequence from the issue directly.The existing child functions in the module now release their resources through an
ExitStackwhen a test fails.The test runs in a child process and is parametrized over the import paths named in the issue: a single import, several imports of one raw handle, and concurrent imports of one UUID. It asserts that the re-import grants read/write access, that a buffer imported through it can be filled, and that no
CUDAWarningis issued.Checklist