You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
cuda.core: a host copy function with the CUhostFn signature for host_launch #3256
host_launch (#2058) accepts a ctypes function pointer declared as ctypes.CFUNCTYPE(None, ctypes.c_void_p). The driver calls such a pointer directly, with no Python frame and no GIL. That is the right path for a host copy that runs in stream order, such as the pipeline in #2058 (comment): each chunk is a device-to-pinned copy followed by a host function that copies the pinned slot into the destination array, while the main thread keeps enqueuing work.
A pure-Python package has nothing to point that pointer at. A ctypes callback built from a Python function always acquires the GIL. The C library's memcpy takes three arguments, not the one void * of CUhostFn. So the package needs a compiled extension of its own for one memcpy.
Proposal
Provide one C host function in cuda.core with the CUhostFn signature. It reads dst, src, and nbytes from the user_data it receives and copies the bytes. It never enters Python, so it never takes the GIL. Expose it in a form that host_launch and the graph callback methods accept as fn unchanged, together with a small helper that packs (dst, src, nbytes) into user_data bytes. cuda.core copies the packed arguments and owns the copy until the function has run, which is the existing user_data contract for bytes.
The pipeline above then needs no extension: per chunk, a copy_to into the pinned slot, then host_launch of the copy function with the packed arguments.
The same function serves #2989, where a stream-ordered release needs an in-package host function.
Design question
The public form of the function. Two candidates:
A module attribute: a ctypes function pointer with the declared CUhostFn prototype. It passes through the existing validation and works in the stream path and the graph path alike. Recommended: it needs no new entry point.
A free function host_copy(stream, dst, src, nbytes) beside launch and host_launch. It would duplicate the capture handling that host_launch and GraphBuilder.callback already have.
Out of scope
A staged device-to-host copy utility with a pinned staging ring. Slot sizing, slot reuse, and backpressure are application policy and belong to the caller, not to the cuda.core foundation. The copy function is the piece that lets a package build such a ring in pure Python.
@Andy-Jost, I'll take this. I agree with option 1. _resolve_host_callback already accepts a CUhostFn ctypes pointer, so a single attribute works with host_launch, all the graph callback methods and update(). A free function would cover only the stream path.
Plan:
cuda.core.HOST_COPY_FN: a ctypes.CFUNCTYPE(None, ctypes.c_void_p) instance that points to a noexcept nogil C function. It reads (dst, src, nbytes) from user_data and calls memcpy.
cuda.core.pack_host_copy_args(dst, src, nbytes) -> bytes: packs the arguments for user_data, under the existing bytes ownership contract.
dst and src are not retained, as with Buffer.copy_to.
nbytes == 0 does nothing. Overlapping ranges raise ValueError.
#2989 needs a different host function, one that enqueues deferred cleanup rather than copying memory, so I'd drop it from this issue.
On driver 615.78.08, every host function in a context runs on one driver thread, in order, regardless of stream. A 1 s callback on one stream delayed callbacks on 15 other streams by 1 s. A second context got its own thread. So a large copy delays every other host function in its context, including Python callbacks. I'll document this and recommend chunk sizes.
Questions:
Should the argument layout (uintptr_t dst, uintptr_t src, size_t nbytes) be public? Then a ring could pre-pack one struct per slot and pass it as an int user_data, which avoids a malloc per chunk. I lean toward public.
Should pack_host_copy_args take only integer addresses, or also Buffer and buffer-protocol objects? Objects would let it check host accessibility and bounds.
Thanks for taking it, @srimanachanta. The plan matches the proposal; answers below.
Name.cuda.core.HOST_COPY_FN fits the package's precedent for module-level singleton objects: LEGACY_DEFAULT_STREAM and PER_THREAD_DEFAULT_STREAM are exported the same way, and the API guidelines want functions to be verbs, which pack_host_copy_args is. Keep both names.
Question 1, the argument layout. Make it public. Document the record as three native-width fields in this order, uintptr_t dst, uintptr_t src, size_t nbytes, with no padding beyond the natural alignment, so an application can pre-pack one record per ring slot and pass its address as an integer user_data, and can read a pack back. pack_host_copy_args then is a convenience over a documented contract, not the only way in.
Question 2, what pack_host_copy_args accepts. Accept integers, Buffer, and buffer-protocol objects. From objects, take the address and the size and check bounds; a bounds error at pack time is the one check that costs nothing and catches the common mistake. Do not check host accessibility with a driver query inside pack_host_copy_args: a pack can happen on a callback thread, for example a Python host callback that enqueues the next chunk, and a driver call is not allowed there. Use only what is known without the driver: a Buffer knows whether its resource is host-accessible, and a buffer-protocol object is host memory by construction. A raw integer is the caller's responsibility; say so in the docstring.
#2989 needs a different host function, as you say; keep it out of this issue.
The per-context host-function thread is worth a paragraph in the docs with the chunk-size advice, since it also serializes the user's own Python callbacks behind a long copy.
@Andy-Jost I built #3269 on top of your #3258 rebased on main. Lmk if you have any feedback, and I will rebase onto main it once #3258 is merged and if any changes are made to #3258.
Activity
srimanachanta commentedon Oct 28, 2026
@Andy-Jost, I'll take this. I agree with option 1.
_resolve_host_callbackalready accepts aCUhostFnctypes pointer, so a single attribute works withhost_launch, all the graph callback methods and update(). A free function would cover only the stream path.Plan:
#2989 needs a different host function, one that enqueues deferred cleanup rather than copying memory, so I'd drop it from this issue.
On driver 615.78.08, every host function in a context runs on one driver thread, in order, regardless of stream. A 1 s callback on one stream delayed callbacks on 15 other streams by 1 s. A second context got its own thread. So a large copy delays every other host function in its context, including Python callbacks. I'll document this and recommend chunk sizes.
Questions:
Andy-Jost commentedon Oct 28, 2026
Thanks for taking it, @srimanachanta. The plan matches the proposal; answers below.
Name.
cuda.core.HOST_COPY_FNfits the package's precedent for module-level singleton objects:LEGACY_DEFAULT_STREAMandPER_THREAD_DEFAULT_STREAMare exported the same way, and the API guidelines want functions to be verbs, whichpack_host_copy_argsis. Keep both names.Question 1, the argument layout. Make it public. Document the record as three native-width fields in this order,
uintptr_t dst,uintptr_t src,size_t nbytes, with no padding beyond the natural alignment, so an application can pre-pack one record per ring slot and pass its address as an integeruser_data, and can read a pack back.pack_host_copy_argsthen is a convenience over a documented contract, not the only way in.Question 2, what
pack_host_copy_argsaccepts. Accept integers,Buffer, and buffer-protocol objects. From objects, take the address and the size and check bounds; a bounds error at pack time is the one check that costs nothing and catches the common mistake. Do not check host accessibility with a driver query insidepack_host_copy_args: a pack can happen on a callback thread, for example a Python host callback that enqueues the next chunk, and a driver call is not allowed there. Use only what is known without the driver: aBufferknows whether its resource is host-accessible, and a buffer-protocol object is host memory by construction. A raw integer is the caller's responsibility; say so in the docstring.#2989 needs a different host function, as you say; keep it out of this issue.
The per-context host-function thread is worth a paragraph in the docs with the chunk-size advice, since it also serializes the user's own Python callbacks behind a long copy.
srimanachanta commentedon Oct 28, 2026
@Andy-Jost I built #3269 on top of your #3258 rebased on
main. Lmk if you have any feedback, and I will rebase onto main it once #3258 is merged and if any changes are made to #3258.