Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions cuda_core/cuda/core/_cpp/rt/DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -305,17 +305,18 @@ one-time function-table fill (`ensure_fn_table()`). Neither may run while a
C++ lock is held.

**The GIL is the outermost lock.** Code that holds a C++ lock (a registry's
mutex, `ipc_import_mutex`, any `std::mutex`) must not acquire or reacquire the
mutex, any `std::mutex`) must not acquire or reacquire the
GIL while the lock is held: no `report_*` or `pw_*` calls, no
`GILAcquireGuard`, and no `GILReleaseGuard` whose destructor runs inside the
locked region. Code that needs a C++ lock and may run with the GIL held
releases the GIL first (`GILReleaseGuard` before `lock_guard`). Otherwise a
thread blocked on the lock while holding the GIL deadlocks with the lock holder
waiting for the GIL (#2840). Collect statuses under the lock and report after it
is released, as `deviceptr_import_ipc` does: `cleanup_in_context` takes an
`after_cleanup` hook that runs once the cleanup is done and before anything that
may run user code, and the deleter passes one that unlocks its
`std::unique_lock`. The registries store `weak_ptr`s,
is released, as `deviceptr_import_ipc` does for a discarded import. A deleter
that must update shared state before user code can run does so in
`cleanup_in_context`'s `after_cleanup` hook, which runs once the cleanup is done
and before anything that may run user code; the IPC pointer deleter releases its
registry entry there. The registries store `weak_ptr`s,
so erasing an entry under a registry lock never runs a deleter.

### Static Initialization and Deadlock Hazards
Expand Down
16 changes: 16 additions & 0 deletions cuda_core/cuda/core/_cpp/rt/REGISTRY_DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,22 @@ Instances: `context_registry`, `stream_registry`, `event_registry`,
`kernel_registry`, and, per `GraphBox`, one registry for its direct child
graphs and one for its nodes.

### Imports that wait for an in-flight release

IPC imports use `ReleasingHandleRegistry`, a variant whose entry outlives its
handle's last reference. Only the owner's deleter removes the entry, with
`release()`, after its last driver call on the resource. `get_or_create()`
waits while the entry under its key has expired, so a new owner never overlaps
the previous owner's cleanup, even when the last reference was dropped by a
dependent handle that the importing thread cannot synchronize with.

Instances: `ipc_pool_registry` maps an imported `CUmemoryPool` to its owner,
because every import of a pool in a process returns the same `CUmemoryPool`
(#3510); the owner's cleanup revokes access and destroys the imports.
`ipc_ptr_cache` maps a `CUmemPoolPtrExportData` descriptor to its imported
pointer, because a re-import before the previous mapping is freed would share
that mapping (nvbug 5570902).

### Reused graph handles

A `GraphHandle` weak reference locks whenever any handle of its hierarchy is
Expand Down
10 changes: 9 additions & 1 deletion cuda_core/cuda/core/_cpp/rt/api.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,10 @@ MemoryPoolHandle get_device_mempool(int device_id);

// Create an owning memory pool handle from an IPC import.
// The file descriptor is NOT owned by this handle (caller manages FD separately).
// When the last reference is released, cuMemPoolDestroy is called automatically.
// Repeat imports of one pool share one handle, because access settings belong to
// the pool rather than to an import (#3510). When the last reference is released,
// access is revoked and cuMemPoolDestroy is called once for every import. An
// import that races that release waits until it is done.
// Returns empty handle on error (caller must check).
MemoryPoolHandle create_mempool_handle_ipc(int fd, CUmemAllocationHandleType handle_type);

Expand Down Expand Up @@ -306,6 +309,11 @@ size_t va_reservation_size(const VaReservationHandle &h) noexcept;
long long live_owned_mempool_count() noexcept;
long long live_va_reservation_count() noexcept;

// Test hook: the owner of an imported pool sleeps this long after its last
// reference is released and before it revokes access, so that a re-import can
// be made to race the release. 0 (the default) disables the delay.
void set_ipc_pool_release_delay_for_testing(unsigned int microseconds) noexcept;

// Map the whole allocation at ptr inside the reservation via cuMemMap and
// apply the allocation's access descriptors. The mapping structurally depends
// on both handles. When the last reference is released, cuMemUnmap is called
Expand Down
79 changes: 78 additions & 1 deletion cuda_core/cuda/core/_cpp/rt/internal.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
#include "types.hpp"
#include "driver_api.hpp"
#include "error.hpp"
#include <condition_variable>
#include <cstddef>
#include <cstdint>
#include <functional>
Expand Down Expand Up @@ -192,7 +193,6 @@ class HandleRegistry {
std::scoped_lock lock(mutex_);
map_.erase(key);
}

Handle
lookup(const Key &key)
{
Expand Down Expand Up @@ -244,4 +244,81 @@ class HandleRegistry {
MapType map_;
};

// ============================================================================
// Handle registry that waits for in-flight releases
//
// Like HandleRegistry, but an entry outlives its handle's last reference: only
// the owner's deleter removes it, by calling release() once its cleanup is
// done. get_or_create() waits while the entry under its key has expired, so a
// new owner is never created while the previous owner is still releasing the
// resource. Used where the driver hands the same resource to repeat imports
// (IPC pools and IPC pointers), so a new owner and an old owner's cleanup
// would act on one driver resource. See REGISTRY_DESIGN.md.
// ============================================================================

template <typename Key, typename Handle, typename Hash = std::hash<Key>>
class ReleasingHandleRegistry {
public:
// Return the live handle registered under key, waiting first while the
// previous owner's release is in flight. Otherwise register and return the
// handle that create() returns; an empty handle is returned unregistered.
// create() runs under the registry's mutex, so it must not acquire the GIL
// or release a handle whose deleter calls release().
template <typename Factory>
Handle
get_or_create(const Key &key, Factory &&create)
{
std::unique_lock lock(mutex_);
for (auto it = map_.find(key); it != map_.end(); it = map_.find(key)) {
if (Handle h = it->second.handle.lock()) {
return h;
}
released_.wait(lock);
}
// Insert before creating, so a failed insertion has nothing to undo.
auto it = map_.try_emplace(key).first;
Handle h;
try {
h = create();
}
catch (...) {
map_.erase(it);
throw;
}
if (h) {
it->second = Entry{h, h.get()};
}
else {
map_.erase(it);
}
return h;
}

// Called by an owner's deleter after its last driver call on the resource:
// erase key's entry if it belongs to the handle that pointed to resource,
// and wake the importers waiting for it.
void
release(const Key &key, const typename Handle::element_type *resource) noexcept
{
{
std::scoped_lock lock(mutex_);
auto it = map_.find(key);
if (it != map_.end() && it->second.resource == resource) {
map_.erase(it);
}
}
released_.notify_all();
}

private:
struct Entry {
std::weak_ptr<typename Handle::element_type> handle;
const typename Handle::element_type *resource = nullptr;
};

std::mutex mutex_;
std::condition_variable released_;
std::unordered_map<Key, Entry, Hash> map_;
};

} // namespace cuda_core::rt::detail
Loading
Loading