cuda.core: define the error handling policy and report failures that cannot be raised by Andy-Jost · Pull Request #2759 · NVIDIA/cuda-python

Andy-Jost · 2026-09-03T00:37:30Z

Description

Closes #2758 (the error handling policy RFC). Builds on #2750, which has merged; this PR is now directly on main.

Writes down cuda.core's failure-handling policy and brings the code into line with it.

Policy (docs):

New docs/source/error_handling.rst: what an exception from cuda.core guarantees (nothing created, caller's context unchanged), how a secondary failure is attached to the exception being raised, how failures that cannot be raised are reported, context restoration failures, sticky errors, interpreter shutdown, and that cuda.core never terminates the process.
New "Failure handling" section in cuda_core/AGENTS.md (contributor rules: raise by default, attach when raising and report only when nothing can be raised, publish before you raise, leak rather than dangle, one reporting channel, no hand-rolled context push/pop, never terminate the process: an internal invariant violation is raised or reported and the resource leaked).
_cpp/DESIGN.md: the two status conventions of the C++ layer (handle factories vs. CUresult-returning helpers; unifying them is cuda.core: return CUresult from C++ handle factories instead of thread-local error state #2760), context-scoped operations, and reporting from non-propagating paths.

Code:

CUDAError and NVRTCError are exported from cuda.core; the docs tell users to catch CUDAError, which previously lived only in a private module.
New public cuda.core.CUDAWarning (a RuntimeWarning), emitted for CUDA errors that cannot be raised. Reports name the resource handle (cuMemFreeAsync(0x...) failed ...) so Python's warning registry does not collapse independent failures of the same call from one call site. The C++ handle layer reports through one helper (report_cuda_error / report_message) that goes through the Python warnings machinery when the interpreter is usable, delivers an escalated warning as an unraisable exception, and falls back to stderr (flushed) otherwise. CUDA_ERROR_DEINITIALIZED is filtered.
Every destroy call made from a deleter is wrapped (pw_*) and reported on failure, including the ones cuda.core: make Device methods use their bound context #2750 left silent (memory pools, green contexts, graphs, graph execs, graphics, linker, user objects, NVRTC/NVVM/nvJitLink handles, file descriptors). The compiler-handle deleters now release the GIL like the CUDA ones.
Secondary failures are attached to the propagating exception instead of reported out of band: a failed context restoration becomes a PEP 678 note on the raised CUDAError (the operation's error stays primary when both fail; the message suffix is used on Python 3.10), and a failed rollback inside a Cython except block is attached to the exception being handled via attach_rollback_failure (a CUDAWarning when nothing is being handled). The thread-local detail is keyed to the status it was recorded for, which narrows misattribution but does not remove it when a caller drops the status; cuda.core: return CUresult from C++ handle factories instead of thread-local error state #2760 replaces this state with explicit status returns. In deleters a restoration failure is a CUDAWarning; a skipped context-sensitive undo is reported with its status instead of leaking silently.
The GIL is the outermost lock in the C++ layer: nothing that holds a C++ mutex may acquire it. deviceptr_import_ipc releases the GIL before taking ipc_import_mutex, holds the mutex across unregister and free in the cache deleter (closing the expired-entry race that Reorder GIL/ipc mutex lock in resource_handles.cpp #2848's concurrent-import test exposed), and reports only after unlocking via cleanup_in_context's after_cleanup hook. DESIGN.md has the rule, a table of which reporting channel to use where, and what pw_ wrappers actually run.
New handle-layer helpers context_get_device and graph_node_set_params; Stream_get_ctx_device and _set_definition_node_params use them instead of hand-rolled push/pop. The node update now publishes its attachment before raising a restoration failure, closing a dangling-owner window.
Device.set_current(ctx) switches with a single cuCtxSetCurrent (a failure leaves the previous context current; works with no context current).
GraphBuilder.__dealloc__ reports a failed cuStreamEndCapture that was silent.
_mr_dealloc_callback warns instead of printing to stderr.
Test hooks cuda.core._resource_handles._set_context_restore_fault_for_testing and _attach_rollback_failure_for_testing; new tests/test_error_handling.py; tests/test_memory.py asserts on CUDAWarning instead of stderr text.
Release notes in docs/source/release/1.3.0-notes.rst.

Follow-ups filed from the review: #2760 (return CUresult from handle factories instead of thread-local err), #2761 (RFC: sticky errors as a BaseException subclass). Looking up raw driver function pointers instead of the cydriver wrappers is deferred to a separate change.

Checklist

New or existing tests cover these changes.
The documentation is up to date with these changes.

🤖 Generated with Claude Code

copy-pr-bot · 2026-09-03T00:37:35Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

Andy-Jost · 2026-09-03T00:47:14Z

-                HANDLE_RETURN(cydriver.cuCtxPopCurrent(&prev_ctx))
-                HANDLE_RETURN(cydriver.cuCtxPushCurrent(curr_ctx))
+                # cuCtxSetCurrent replaces the top of the thread's context stack
+                # in one driver call (or binds ctx when nothing is current), so
+                # a failure leaves the previous context current instead of
+                # leaving the thread with no context, as a failed pop-then-push
+                # would.
+                HANDLE_RETURN(cydriver.cuCtxSetCurrent(curr_ctx))


I'm not certain whether the two-call sequence here is needed for a reason I don't see. It would be good to confirm this change.

Andy-Jost · 2026-09-03T03:19:12Z

/ok to test

Andy-Jost · 2026-09-03T03:25:31Z

/ok to test

github-actions · 2026-09-03T03:40:21Z

Doc Preview CI
🚀 View preview at https://nvidia.github.io/cuda-python/pr-preview/pr-2759/
https://nvidia.github.io/cuda-python/pr-preview/pr-2759/cuda-core/
https://nvidia.github.io/cuda-python/pr-preview/pr-2759/cuda-bindings/
https://nvidia.github.io/cuda-python/pr-preview/pr-2759/cuda-pathfinder/
Preview will be ready when the GitHub Pages deployment is complete.

mdboom

Left some comments below.

Looking at the various usages below, I think we should add to this the recommendations (in AGENTS.md etc) that C++ should call CUDA APIs by looking up the function pointers using _inspect_function_pointers (which returns the unvarnished underlying CUDA function) rather than __pyx_capi__ (which returns the cython layer). Calling the cython layer requires handling/clearing Python exceptions which is (a) easily forgotten and (b) may not even be possible in all contexts. That would still take advantage of any logic the internal layer implements to support multiple CTK versions, so is preferable to just linking directly to the CTK.

mdboom · 2026-09-03T12:50:05Z

+Some functions return a `CUresult` directly instead of a handle (for example
+`context_synchronize`, `context_get_device`, `graph_node_set_params`). Their
+callers `HANDLE_RETURN` the value.


Maybe a quick note about the deciding factor behind why some functions don't raise exceptions?

I added a note and also filed a follow-up issue: #2760

The discrepancy stems from a historical progression. Handles from this module were created as h = create_handle(...) followed by a null check if not h:. But over time, as more non-handle functions were added, they naturally returned CUresult and this discrepancy appeared.

It's probably best to adopt the driver convention uniformly, so handles would be created as status = create_handle(&h, ..) instead. Returning the handle would make sense if the exception-raising path were viable, but we settled on always raising exceptions from Cython rather than C++, usually via HANDLE_RETURN, so returning handles really has no value anyway.

mdboom · 2026-09-03T13:01:23Z

+- cleanup performed after an operation has already failed, such as rolling back
+  a partially built graph node or restoring the caller's CUDA context.


While I agree with the first 2 cases, I'm not sure the third case must be that way (unless there is some CUDA reason for it that I am missing). I think exception chaining with raise ... from by appropriate here. It's intended for "while this exception was being handled, this other exception happened".

I adjusted the wording here and implemented a fix based on Python 3.11's add_note.

One thing to point out is that I coded it so that when a context-scoped operation and the context restoration both fail, cuda.core raises the operation failure as the primary error and attaches the failed restoration as a note. An example of such a failure might read as follows:

Traceback (most recent call last): ... cuda.core._utils.cuda_utils.CUDAError: CUDA_ERROR_OUT_OF_MEMORY: The API call failed because it was unable to allocate enough memory ... the calling thread's CUDA context (0x55d0...) could not be restored after this failure (cuCtxSetCurrent: CUDA_ERROR_INVALID_CONTEXT); context 0x7f3a... is now current. Call Device.set_current() before issuing further CUDA work on this thread

Raising chained exceptions instead would keep the same chronological order in the traceback (out of memory first, then "During handling of the above exception, another exception occurred:", then the restoration failure), but the restoration failure would become the exception that propagates: except CUDAError as e would catch CUDA_ERROR_INVALID_CONTEXT. Open to your suggestion here.

There are basically 3 options in 3.11 Python and later for handling multiple exceptions: add_note, exception chaining and exception groups. I agree add_note is probably the right one for this case. Exception chaining may be relevant for us in other situations, where the later exception really supercedes the first one. Exception groups, after a reminder read, are really about async / concurrency -- where multiple exceptions happened in an undefined order as part of async work of a larger chain. I don't think that's likely to ever apply to cuda-core.

Agreed. I wonder if exception groups could be useful in conjunction with graphs. Say a graph recapture encounters 10 separate errors while updating nodes, for example.

mdboom · 2026-09-03T13:07:25Z

+``cuda.core`` raises these errors like any other and does not attempt to
+recover from them. It does not terminate the process for you: the exception
+carries the Python traceback of the call that observed the fault, and your
+application decides how to shut down.


Since the intent here is that the user should basically stop and there is no legitimate way to recover, this /might/ be a use case for an exception that inherits from BaseException rather than Exception. Those are not caught by default in bare try: ... except: ... so naive code that wants to "if anything went wrong, try again" would still get truncated. It's still possible to catch these BaseException exceptions, but it adds a little an extra speedbump that might be desirable.

This makes sense to me. Filed #2761 to discuss.

As an aside to this conversation, I don't see a public way of importing CUDAError. The current placement appears to be from cuda.core._utils.cuda_utils import CUDAError.

My reading of this section is that my application is expected to catch CUDAError. If so, it should either be promoted to a public path or I'd be at least one vote for more precise wording here.

That's a good point. I added CUDAError and NVRTCError to the top-level exports.

mdboom · 2026-09-03T13:27:09Z

+    cdef const char* desc
+    # A context-scoped helper in the handle layer may have recorded why this
+    # status needs more explanation (e.g. the caller's context was not restored).
+    cdef const char* detail = take_last_error_detail()
    name_err = cydriver.cuGetErrorName(error, &name)
    if name_err != cydriver.CUresult.CUDA_SUCCESS:
        raise CUDAError(f"UNEXPECTED ERROR CODE: {error}")
+    desc_err = cydriver.cuGetErrorString(error, &desc)
    with gil:
+        suffix = f" ({detail.decode()})" if detail != NULL else ""
        # TODO: consider lower this to Cython
        expl = DRIVER_CU_RESULT_EXPLANATIONS.get(int(error))
        if expl is not None:
-            raise CUDAError(f"{name.decode()}: {expl}")
-    cdef const char* desc
-    desc_err = cydriver.cuGetErrorString(error, &desc)
-    if desc_err != cydriver.CUresult.CUDA_SUCCESS:
-        raise CUDAError(f"{name.decode()}")
-    raise CUDAError(f"{name.decode()}: {desc.decode()}")
+            raise CUDAError(f"{name.decode()}: {expl}{suffix}")
+        if desc_err != cydriver.CUresult.CUDA_SUCCESS:
+            raise CUDAError(f"{name.decode()}{suffix}")
+        raise CUDAError(f"{name.decode()}: {desc.decode()}{suffix}")


This is a classic use case for an exception group, released in Python 3.11. The idea is that anytime you want to attach information about an exception to another exception, it should be done in a standardized way so the Python programmer has a known way to pick it back apart and then except* will work etc.

For Python 3.10, I think we still need something like this as a fallback, but since that will be EOL in ~6 weeks, we should have an implementation that uses exception groups on 3.11 as the canonical implementation now.

Would an exception group break existing code that catches CUDAError, because the group has a different type? My agent is suggesting the Python 3.11+ add_note feature.

+1 to add_note

mdboom · 2026-09-03T13:32:15Z

+- **Rollback failure**: the original exception propagates; the failed rollback
+  is reported out of band (or chained with `raise ... from` when a second
+  exception must be raised). Bare `except:` is acceptable only for
+  rollback-then-`raise` blocks.


Should this be:

Suggested change

- **Rollback failure**: the original exception propagates; the failed rollback

is reported out of band (or chained with `raise ... from` when a second

exception must be raised). Bare `except:` is acceptable only for

rollback-then-`raise` blocks.

- **Rollback failure**: the original exception propagates; the failed rollback

is chained with `raise ... from` when possible, otherwise reported out-of-band. Bare `except Exception:` is acceptable only for

rollback-then-`raise` blocks.

mdboom · 2026-09-03T13:32:59Z

+  must go through a single helper that writes a diagnostic (call, CUDA error,
+  invariant, "please report") to stderr before aborting, must never trigger


...and a Python traceback using the stdlib faulthandler...

mdboom · 2026-09-03T13:39:53Z

-    ScopedCurrentContext& operator=(const ScopedCurrentContext&) = delete;
+    template <typename... Args>
+    auto operator()(Args&&... args) const noexcept {
+        auto status = Function(std::forward<Args>(args)...);


The function pointer here, since it comes from cydriver.__pyx_capi__ will never be non-NULL. However, the underlying pointer to the actual CUDA API function may be NULL, in which case a Python FunctionNotFoundError is raised and CUDA_ERROR_NOT_FOUND is returned. This may only be a latent issue right now because we only ever use functions that always resolve (exist in every version of CUDA we support).

We should either:

Check for CUDA_ERROR_NOT_FOUND here and call PyErr_Clear to not keep the ignored exception in the global state. (But that requires having a GIL and a Python interpreter in a working state.)

Look up underlying CUDA function pointer instead using _inspect_function_pointers() rather than __pyx_capi__. Then NULL-check the function pointer either when looking it up or right before calling it.

+1 though I'm planning to deal with these issues in a follow-up to keep this PR's scope under control.

mdboom · 2026-09-03T13:44:19Z

+        err = p_cuGreenCtxStreamCreate
+            ? p_cuGreenCtxStreamCreate(&stream, as_cu(h_green), flags, priority)
+            : CUDA_ERROR_NOT_SUPPORTED;


This is a concrete example of the function pointer issue I mentioned above:

p_cuGreenCtxStreamCreate will be non-NULL for any cuda-bindings that was built with that symbol, but the underlying pointer it calls may be NULL if the CTK or driver underneath is too old. This doesn't actually trap for that case, and it leaves a Python exception set that is ignored but never cleared.

I agree with the assessment. I'd like to defer this issue and deal with two issues together in a follow-up:

Require a new-enough cuda.bindings.

Call raw driver functions directly rather than use Cython wrappers

The first ensures these p_ pointers are never null. The second corrects the bugs you're referring to.

Andy-Jost · 2026-09-03T16:59:05Z

/ok to test

Andy-Jost · 2026-09-03T20:20:53Z

            return;
        }
+    }
+    std::fprintf(stderr, "%s\n", message);


flush() here?

Andy-Jost · 2026-09-04T21:49:56Z

/ok to test

…cannot be raised Write down how cuda.core handles CUDA failures (docs/source/error_handling.rst for users, a "Failure handling" section in AGENTS.md and _cpp/DESIGN.md for contributors) and bring the code into line with it: - Add cuda.core.CUDAWarning, emitted for CUDA errors that cannot be raised (destructors, CUDA callbacks, cleanup after an earlier failure). The C++ handle layer reports through one helper that uses the Python warnings machinery when the interpreter is usable, delivers an escalated warning as an unraisable exception, and falls back to stderr otherwise. CUDA_ERROR_DEINITIALIZED is not reported. - Wrap every destroy call made from a deleter (pw_*) so its failure is reported instead of discarded, including memory pools, green contexts, graphs, graph execs, graphics resources, the linker, user objects, the NVRTC/NVVM/nvJitLink handles and file descriptors; release the GIL around the compiler-handle destroys like the CUDA ones. - When the caller's context cannot be restored after a successful operation, undo the creation and raise a CUDAError that says which context is current; report the same failure as a warning in deleters; report a skipped context-sensitive undo instead of leaking silently. - Add context_get_device and graph_node_set_params so Stream_get_ctx_device and _set_definition_node_params stop hand-rolling cuCtxPush/Pop/SetCurrent. The node update now publishes its attachment before raising a restoration failure, closing a window that left the node referencing released owners. - Device.set_current(ctx) switches with a single cuCtxSetCurrent, so a failure leaves the previous context current and the call works without one. - Report failed cuStreamEndCapture in GraphBuilder.__dealloc__ and failed child-graph rollbacks; warn from _mr_dealloc_callback instead of printing. - Add a test hook that makes the next context restoration fail, tests for the policy, and release notes for 1.3.0. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

The "Errors and warnings" section was inserted between the texture classes and the texture option dataclasses, which moved OpaqueArrayOptions, MipmappedArrayOptions and TextureObjectOptions under cuda.core in the docs index and failed test_api_docs_consistency on every CI platform. Place the section after the texture section instead. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…notes Review follow-ups on the error-handling policy: - A failure that happens while an exception is being raised is no longer reported out of band. When both an operation and the restoration of the caller's context fail, the operation's CUDAError is raised with the restoration failure attached; when only the restoration fails, its error is raised with the context explanation attached. The attachment is a PEP 678 note on Python 3.11+ and is appended to the message on 3.10. The thread-local detail is keyed to the status it was recorded for, so it cannot attach to an unrelated error if that status is never raised. - A failed rollback inside a Cython `except` block is attached to the exception being handled through note_or_report_cuda_error(), which falls back to a CUDAWarning when nothing is being handled or notes are unavailable. - Reporting stays reserved for destructors and CUDA callbacks; CUDAWarning's docstring and the docs say so. - DESIGN.md explains the two status conventions of the C++ layer (handle factories use thread-local err, everything else returns CUresult) and the abort-helper guidance in AGENTS.md asks for a faulthandler-style traceback. - Drop the release-relative "in this release" wording from the stable docs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…back Rebased onto the reviewed head of NVIDIA#2750. Adjustments the rebase needed: - The review's warning for an undo skipped after a failed context restoration is routed through report_cuda_error(), so it carries the CUDA status and becomes a CUDAWarning like every other non-raising report. - invoke_in_context and invoke_in_context_or_undo now reject empty handles themselves, so context_get_device drops its own guard like the other helpers did; enter_context's no-op for empty handles is documented as used only by graph_node_set_params. - _SynchronousMemoryResource moved to its own module; the error-handling test imports it from there. The review's two teardown tests asserted that stderr stayed empty; under the policy a teardown failure is a CUDAWarning, so they assert that no CUDAWarning is issued instead (and are marked thread_unsafe because warning capture is process-global). - report_message() flushes stderr after its last-resort fprintf, so the text is not lost if the process dies right after (review comment). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Andy-Jost · 2026-09-05T15:16:04Z

/ok to test

…policy # Conflicts: # cuda_core/cuda/core/_memory/_buffer.pyx # cuda_core/tests/test_memory.py

… stderr The host-only Buffer tests from NVIDIA#2773 asserted that nothing containing "Warning" reached stderr. Under the error handling policy a teardown failure is a CUDAWarning, not stderr text, so that assertion no longer checks anything. Use assert_no_cuda_warning() around allocate/close instead (marked thread_unsafe, as warning capture is process-global). The spawned-process variant checks inside the child, since warnings do not cross processes; a failure surfaces as the non-zero exit code the parent already asserts on. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…policy # Conflicts: # cuda_core/docs/source/release/1.3.0-notes.rst

The policy text reserved std::abort for an internal invariant violation and specified how such a helper would have to behave. The decision is that cuda.core never terminates the process: an internal invariant violation is raised as a RuntimeError where an exception can propagate, reported as a CUDAWarning where it cannot, and the affected resource is leaked. Users who want fail-fast behavior escalate the warning category themselves. An implicit abort (an exception escaping noexcept code) remains a bug, not a policy choice. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Andy-Jost · 2026-09-11T20:52:23Z

@kkraus14 @leofang you might want to review and approve the proposed error policy.

brandon-b-miller · 2026-09-12T03:17:56Z

+    PyObject* category = warning_category.load(std::memory_order_acquire);
+    if (category && Py_IsInitialized() && !py_is_finalizing()) {
+        GILAcquireGuard gil;
+        if (gil.acquired()) {


This line of code led me to find a deadlock that can occur inside the IPC code which should maybe be fixed separately/is not due directly to this PR, however maybe motivates a related todo here.

TLDR code in this layer should never block on the gil while holding a c++ lock and we should consider writing that policy down in DESIGN.md.

We're changing from a function that doesn't need the gil to one that does need the gil, and introducing a number of additional places that it can be called. So the natural question becomes "is it safe to run python in all the places that newly require the gil"?

Auditing those turned up #2840.

Let's merge #2848 first and then integrate into this PR. In addition to the lock reorder, we need to adjust the call to pw_cuMemFreeAsync in deviceptr_import_ipc. It should call the non-reporting version and then issue the report after ipc_import_mutex is released. I also updated DESIGN.md with deadlock-avoidance rules relating to the GIL and C++ locks in this layer.

brandon-b-miller · 2026-09-12T03:18:01Z

+            PyObject *pending_type, *pending_value, *pending_tb;
+            PyErr_Fetch(&pending_type, &pending_value, &pending_tb);
+#endif
+            if (PyErr_WarnEx(category, message, 1) != 0) {


This appears to contain a small bug that affects what users see when getting multiple similar but really independent failures.

TL;DR Python's warning registry cache keys on (message text, category, lineno), and none of those vary across independent resources failing the same call. so you can get at most one warning each.

Here I fabricate two genuinely independent cleanup failures of the same kind.

Create some buffers on the default stream. cuda-core bookkeeping holds a reference to the device context that was current, since the token doesn't hold that information.

import warnings from cuda.core import CUDAWarning, Device, DeviceMemoryResource, DeviceMemoryResourceOptions from cuda.core._resource_handles import _set_context_restore_fault_for_testing from cuda.core._stream import default_stream from cuda.core._utils.cuda_utils import driver, handle_return INVALID_CONTEXT = int(driver.CUresult.CUDA_ERROR_INVALID_CONTEXT) device = Device() device.set_current() device_context = int(device.context.handle) mr = DeviceMemoryResource(device, DeviceMemoryResourceOptions(max_size=2 << 20)) # Default-stream allocations record the allocating context, so freeing them with # no context current must switch to it and switch back. buffers = [mr.allocate(256, stream=default_stream()) for _ in range(2)] delivered = [] warnings.showwarning = lambda msg, cat, *a, **k: ( delivered.append(str(msg)) if issubclass(cat, CUDAWarning) else None )

No filters installed, so this is stock default behavior.

Now, pop so that the calling thread has no current context at all.

handle_return(driver.cuCtxPopCurrent())

So when we free the buffers now, the contract IIUC is "switch to the context to do the free, then switch back". So we're going from no context, to the device context, then back to no context. So we use the handy dandy hook to inject the "fake" failure here when freeing the buffers, and confirm that:

a context switch actually happened

the thread was left stranded on the device context and couldn't switch back

for buffer in buffers: _set_context_restore_fault_for_testing(INVALID_CONTEXT) buffer.close() # Driver-confirmed: a failed restoration leaves the device context current. assert int(handle_return(driver.cuCtxGetCurrent())) == device_context handle_return(driver.cuCtxSetCurrent(driver.CUcontext(0))) # RESET!

So both teardowns really do fail, confirmed against the driver rather than inferred from the machinery under test. We should get a report from each:

cuMemFreeAsync failed while restoring the caller's context: CUDA_ERROR_INVALID_CONTEXT: invalid device context

But only one appears

print(f"restoration failures incurred: {len(buffers)}") print(f"CUDAWarnings delivered: {len(delivered)}") for text in delivered: print(f" {text}") # restoration failures incurred: 2 # CUDAWarnings delivered: 1

The text has no resource identity, the category is constant, and stacklevel=1 pins lineno to the release site, so nothing in the key varies between the two failures and the second is a cache hit.

I'm not really sure what the fix should be here. Putting the pointer in the message disambiguates what resource but I feel like it works against what the caching is intended to prevent.

Good catch. It seems reasonable to me to put the resource ID into the message to avoid de-duplication. Each discrete leak is a separate event that should be reported. Will add this to the next commit.

brandon-b-miller · 2026-09-12T03:18:24Z

+
+const char* take_last_error_detail(CUresult status) noexcept {
+    if (!last_error_detail[0] || status != last_error_detail_status) {
+        return nullptr;


So IIUC the design of this PR is to stash "your context cant be restored" in a thread local keyed on CUresult. The problem I see here is that you may need to make sure it gets consumed by _check_driver_error via HANDLE_RETURN.

I found at least one place, Event._init, where we dont return a status and return an empty handle instead. So you could get a RuntimeError if something goes wrong at this point and I assume the thread is silently stuck on the wrong context. That is one thing the design is intended to prevent.

As a side effect it may be that the fact there was a failure could get misattributed to a later different failure. Trying to get a repro of this.

On this branch:

from cuda.core import Device from cuda.core._resource_handles import _set_context_restore_fault_for_testing from cuda.core._utils.cuda_utils import CUDAError, driver, handle_return INVALID_VALUE = int(driver.CUresult.CUDA_ERROR_INVALID_VALUE) device = Device() device.set_current() device_context = int(device.context.handle) # No context current, so creating an event must switch to the device's context # and switch back. Make that restoration fail with a very ordinary status. previous = handle_return(driver.cuCtxPopCurrent()) _set_context_restore_fault_for_testing(INVALID_VALUE) print("step 1 -- create_event() while the restoration fails") try: device.create_event() print(" no exception (unexpected)") except Exception as exc: print(f" raised : {type(exc).__name__}: {exc}") print(f" notes : {getattr(exc, '__notes__', None)}") _set_context_restore_fault_for_testing(0) stranded = int(handle_return(driver.cuCtxGetCurrent())) == device_context print(f" thread is now stranded on the device's context: {stranded}") print(" ...and was told nothing about it.") print() print("step 2 -- an unrelated failing device-attribute query") try: handle_return(driver.cuDeviceGetAttribute(99999, 0)) print(" succeeded (unexpected)") except CUDAError as exc: print(f" raised : {str(exc).split(':')[0]}") for note in getattr(exc, "__notes__", []): print(f" note : {note}") if any("could not be restored" in n for n in getattr(exc, "__notes__", [])): print() print(" >>> a context-restoration explanation was attached to an error") print(" >>> that has nothing to do with contexts.") handle_return(driver.cuCtxSetCurrent(driver.CUcontext(0))) handle_return(driver.cuCtxPushCurrent(previous))

step 1 -- create_event() while the restoration fails raised : RuntimeError: Failed to create CUDA event notes : None thread is now stranded on the device's context: True ...and was told nothing about it. step 2 -- an unrelated failing device-attribute query raised : CUDA_ERROR_INVALID_VALUE note : the calling thread's CUDA context (0) could not be restored; context 0x5eb9dbbe7640 is now current. Call Device.set_current() before issuing further CUDA work on this thread >>> a context-restoration explanation was attached to an error >>> that has nothing to do with contexts.

Confirmed, thanks for the analysis. I'd like to defer this to #2760, where I added a note to cover the detail buffer. We should do away with the out-of-band thread-local status values and instead return everything directly. Rather than h = create_handle(), we'd write status = create_handle(&h), and for operations that have a context restoration status, I'd add a separate output argument to capture it.

brandon-b-miller · 2026-09-14T14:01:09Z

        status = std::invoke(std::forward<Fn>(operation), std::forward<Args>(args)...);
        if (status != CUDA_SUCCESS) {
-            warn_on_cuda_error(name, status);
+            report_cuda_error(name, status);


I think it might be worth considering moving this to after the exit context below. This path emits a CUDAWarning so the user's filters and showwarning fire here. Especially when dealing with cuda-python I'd say its not out of the question that they have some kind of handling code that interacts with CUDA, lets make sure its the context they expect in the case where restoration succeeds at least.

brandon-b-miller · 2026-09-14T14:22:29Z

+Process termination
+-------------------
+
+``cuda.core`` never terminates the process. A CUDA error is raised, or reported


Not actionable but an FYI, I had an agent try pretty hard to find ways of violating this statement locally, it tried a bunch of interesting tricks and couldn't break it.

brandon-b-miller

Left a few comments on the diff directly. A few smaller things that accumulated looking through this:

I think that p vs pw deserves a little louder distinction that the latter can run arbitrary user python code. This is the main entrypoint through which I was able to shake anything apart.
the PR adds report_cuda_error/report_message, note_or_report_cuda_error, and last_error_detail/take_last_error_detail. But there's no concrete section I can look at to know where and when to pick one. Indeed the missing Event site is the result of picking "none of these" it would seem.

cleanup_in_context() reported an activation or operation failure before it switched back to the caller's context. A CUDAWarning runs user code (warning filters, showwarning), so that code observed the cleanup context instead of the caller's. Emit both reports after the restoration attempt; the report order and the return value are unchanged. Review follow-up on NVIDIA#2759. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…te the detail limitation Review follow-ups on NVIDIA#2759: - Cleanup reports name the resource handle ("cuMemFreeAsync(0x...) failed ..."). Python's warning registry collapses repeated warnings with identical text from one call site, so two independent resources failing the same call from the same line produced a single CUDAWarning. The pw_* wrappers name their first argument, cleanup_in_context() takes the handle explicitly, and the Buffer deallocation callback names the pointer. A test releases two buffers under an injected restoration failure and expects two reports. - CUDAError and NVRTCError are importable from cuda.core; the error handling page told users to catch CUDAError but it lived in a private module. Both classes gained docstrings. - DESIGN.md and the docs no longer claim that keying the thread-local detail to its status prevents misattribution; a caller that drops the status leaves it behind for a later error with the same code. NVIDIA#2760 removes the thread-local state. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

deviceptr_import_ipc() took ipc_import_mutex and only then released the GIL, so a thread blocked on the mutex while holding the GIL deadlocked with the holder waiting to reacquire it at scope exit (NVIDIA#2840). It also called the pw_ wrapper for the discard path under the mutex, and the wrapper acquires the GIL to emit a CUDAWarning. Release the GIL before taking the mutex, keep lookup, import and registration under the mutex so a descriptor is never imported twice, discard with the raw driver call, and report a failed discard only after the lock is released. DESIGN.md states the rule: the GIL is the outermost lock; nothing that holds a C++ lock may acquire or reacquire it. The guard reorder is the same fix as NVIDIA#2848, which also adds the regression test for the deadlock; this change keeps that reorder and adds the deferred report, so whichever lands second resolves the overlapping hunk in its favor. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Review follow-ups on NVIDIA#2759: - DESIGN.md gains a table that picks the reporting channel by situation (a path that can raise, an except block whose rollback failed, a deleter, a Python destructor path, a CUDA callback thread), with the thread-local mechanisms marked transitional pending NVIDIA#2760, and a section on p_ versus pw_: a pw_ wrapper acquires the GIL on failure and runs user Python, so it is never used while a C++ lock is held. Python exceptions raised by that code never become C++ exceptions; nothing on the report path may allocate or throw. CUDA callback threads do nothing that needs the GIL; Py_AddPendingCall is how work leaves them. - AGENTS.md gets the same two rules in short form. The header comments on the reporting functions and on WarnOnFailure say what pw_ runs. - note_or_report_cuda_error is renamed attach_rollback_failure: callers are in one situation (a rollback failed while an exception is in flight) and should not have to know the note-or-warning mechanism. The test hook follows. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Brandon's concurrent-import test (NVIDIA#2848) crashes on CUDA 12.9 once the GIL reorder lets the importers run: the IPC pointer cache's deleter did not take ipc_import_mutex, so a concurrent importer that found the entry expired while the deleter was still freeing re-imported the allocation, got a duplicate pointer to the same mapping (nvbug 5570902), and the first cuMemFreeAsync unmapped it for both. The deleter now releases the GIL, then holds the mutex across unregister and free. The cleanup report emits a CUDAWarning, which acquires the GIL and runs user code, so it must not run under the mutex: cleanup_in_context() gains an overload with an after_cleanup hook, called unconditionally once the cleanup and the context restoration are done and before anything that may run user code, and the deleter passes one that unlocks its std::unique_lock. The deallocation context is resolved before the lock for the same reason. Companion to the main-side fix pushed to NVIDIA#2848. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Andy-Jost · 2026-09-14T21:09:30Z

Thanks @brandon-b-miller for the detailed review. I think I've addressed everything in the latest push. In brief:

Set aside the out-of-band thread-local error state for a follow-up in cuda.core: return CUresult from C++ handle factories instead of thread-local error state #2760, which now also covers the detail buffer.
Cleanup warnings name the resource handle, so independent failures are no longer collapsed by the warning registry.
Cleanup warnings fire after the caller's context is restored.
CUDAError and NVRTCError are exported from cuda.core.
Updated the docs and code for the lock-ordering issues you found; Reorder GIL/ipc mutex lock in resource_handles.cpp #2848 will merge before this does.
Documented which reporting function to use where, and renamed one to attach_rollback_failure to make its role clear.

Ready for re-review.

github-actions Bot added the cuda.core Everything related to the cuda.core module label Sep 3, 2026

Andy-Jost commented Sep 3, 2026

View reviewed changes

Andy-Jost added documentation Improvements or additions to documentation enhancement Any code-related improvements labels Sep 3, 2026

Andy-Jost force-pushed the ajost/error-handling-policy branch from 8719151 to 5e8f7e9 Compare September 3, 2026 03:24

Andy-Jost added this to the cuda.core 1.3.0 milestone Sep 3, 2026

Andy-Jost added the P1 Medium priority - Should do label Sep 3, 2026

mdboom reviewed Sep 3, 2026

View reviewed changes

This was referenced Sep 3, 2026

cuda.core: return CUresult from C++ handle factories instead of thread-local error state #2760

Open

cuda.core: should sticky CUDA errors derive from BaseException? #2761

Open

Andy-Jost requested review from leofang and mdboom September 3, 2026 16:59

Andy-Jost commented Sep 3, 2026

View reviewed changes

Andy-Jost force-pushed the ajost/error-handling-policy branch from 7a632bb to 9773124 Compare September 4, 2026 21:48

Andy-Jost and others added 4 commits September 5, 2026 08:05

Andy-Jost force-pushed the ajost/error-handling-policy branch from 9773124 to c5399ce Compare September 5, 2026 15:07

Andy-Jost requested a review from juenglin September 5, 2026 15:37

Andy-Jost marked this pull request as ready for review September 5, 2026 15:38

Andy-Jost self-assigned this Sep 5, 2026

Merge remote-tracking branch 'origin/main' into ajost/error-handling-…

07e2f6a

…policy # Conflicts: # cuda_core/cuda/core/_memory/_buffer.pyx # cuda_core/tests/test_memory.py

This was referenced Sep 8, 2026

cuda.core tests: memory_ipc/test_errors.py is not thread-safe under pytest-run-parallel (SEGV in cuMemPoolDestroy, shared harness state) #2784

Closed

clear_mempool_peer_access() never takes effect: nvbug 5698116 workaround has been a no-op since #1368 #2807

Open

Andy-Jost and others added 2 commits September 11, 2026 12:43

Merge remote-tracking branch 'origin/main' into ajost/error-handling-…

48754fb

…policy # Conflicts: # cuda_core/docs/source/release/1.3.0-notes.rst

Andy-Jost commented Sep 11, 2026

View reviewed changes

This was referenced Sep 11, 2026

cuda.core: rename _resource_handles to _rt and split its C++ into _cpp/rt/ [stacked on #2759] #2837

Draft

cuda.core: fix pool setup and builder teardown under stream capture #2838

Draft

brandon-b-miller mentioned this pull request Sep 12, 2026

[BUG]: Buffer.from_ipc_descriptor deadlocks between ipc_import_mutex and the GIL #2840

Closed

1 task

brandon-b-miller reviewed Sep 12, 2026

View reviewed changes

brandon-b-miller reviewed Sep 14, 2026

View reviewed changes

Andy-Jost added a commit to Andy-Jost/cuda-python that referenced this pull request Sep 14, 2026

Merge PR NVIDIA#2759 (error-handling policy) into the _rt refactor base

4eb30a0

Andy-Jost and others added 3 commits September 14, 2026 11:19

Andy-Jost mentioned this pull request Sep 14, 2026

Reorder GIL/ipc mutex lock in resource_handles.cpp #2848

Merged

Andy-Jost and others added 2 commits September 14, 2026 13:56

Andy-Jost requested a review from brandon-b-miller September 14, 2026 21:10

		- cleanup performed after an operation has already failed, such as rolling back
		a partially built graph node or restoring the caller's CUDA context.

		must go through a single helper that writes a diagnostic (call, CUDA error,
		invariant, "please report") to stderr before aborting, must never trigger

Uh oh!

Conversation

Andy-Jost commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

copy-pr-bot Bot commented Sep 3, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Andy-Jost commented Sep 3, 2026

Uh oh!

Andy-Jost commented Sep 3, 2026

Uh oh!

github-actions Bot commented Sep 3, 2026

Preview will be ready when the GitHub Pages deployment is complete.

Uh oh!

mdboom left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Andy-Jost Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Andy-Jost commented Sep 3, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Andy-Jost commented Sep 4, 2026

Uh oh!

Andy-Jost commented Sep 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brandon-b-miller Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Andy-Jost commented Sep 3, 2026 •

edited

Loading

Andy-Jost Sep 3, 2026 •

edited

Loading

brandon-b-miller Sep 12, 2026 •

edited

Loading