Repository navigation
Description
A few places in the repo allocate the NVRTC log buffer as log = b" " * logSize and pass it to nvrtcGetProgramLog. When a compile succeeds with an empty log, logSize is 1. b" " * 1 does not make a new object: it returns CPython's cached one-byte b" ", and NVRTC then writes its terminating NUL into that shared object. After that, every b" " in the process is b"\x00", including the literal, " ".encode() and bytes([32]), and b"a b".split(b" ") no longer splits.
These read the log after every compile, including successful ones:
benchmarks/cuda_bindings/runner/runtime.py,compile_cubin()cuda_bindings_13/tests/legacy_api/test_legacy_kernelParams.py,common_nvrtc()
The same line is also in cuda_bindings_12/tests/test_kernelParams.py (common_nvrtc()) and cuda_bindings_12/examples/extra/jit_program_test.py (main()). I have not run those two against cuda-bindings 12.9, so I left them out of the change below; I can add them if you want all four fixed together.
Not affected: cuda_bindings_12/examples/common/common.py only reads the log after a failed compile, where the log is longer than one byte. The cuda.core helpers in _utils/cuda_utils.pyx only allocate when logsize > 1. The b" " * n PTX/CUBIN and LTO-IR data buffers are never one byte long, so they always get a fresh object.
How I found it: I started from NVIDIA-NeMo/Speech#15790, which reports this pattern in NeMo's run_nvrtc(), and searched this repo for b" " * buffers handed to nvrtcGetProgramLog. The places above are the ones that read the log unconditionally.
Repro: I loaded each file unmodified (from main at 0f4983b; the lines are the same on current main), called the function with a trivial kernel, extern "C" __global__ void k(int *p) { if (p) *p = 1; }, and printed the state of b" " before and after the call. nvrtcGetProgramLogSize returned 1 in every run.
| file | b" " after |
b"a b".split(b" ") after |
|---|---|---|
benchmarks/cuda_bindings/runner/runtime.py (compile_and_load()) |
b'\x00' |
[b'a b'] |
cuda_bindings_13/tests/legacy_api/test_legacy_kernelParams.py (common_nvrtc()) |
b'\x00' |
[b'a b'] |
same two files with bytearray(logSize) |
b' ' |
[b'a', b'b'] |
" ".encode() and bytes([32]) returned b'\x00' too after the unmodified calls.
Expected: b" " stays b" ", and b"a b".split(b" ") returns [b'a', b'b'].
Environment: Linux aarch64 with an H100-class GPU, driver 595.58.03, Python 3.12.3, cuda-bindings 13.5.0, cuda-pathfinder 1.9.0, numpy 2.5.5, NVRTC 13.2 from the CUDA 13.2 toolkit.
Where this can show up in practice (my estimate; I have not seen a field report):
common_nvrtc()is the kind of helper people copy into their own projects. In a copied version, whatever else that process does withb" "afterwards breaks silently: splitting or joining byte strings, padding, building protocol messages. Nothing raises; the results are just wrong.- In the test suite, every test that runs after a kernelParams test in the same pytest process sees the corrupted
b" ". A failure would then show up in an unrelated test, or a check that compares againstb" "could pass when it should not. - The benchmark runner compiles every benchmark kernel in one process, so anything later in a run that formats or parses bytes with
b" "is affected.
Fix: allocate the buffer with bytearray(logSize). nvrtcGetProgramLog accepts a bytearray, and log.decode() works the same. With that change b" " is unchanged in both runs above.
CONTRIBUTING asks contributors to coordinate before implementing, so I have not opened a PR. The two-line change is on a branch in my fork: 0z5a@3bc7c6d (branch nvrtc-log-bytearray). If this direction looks right, could you assign this to me? I'll then open the PR.
Activity