Skip to content

[BUG]: b" " * logSize log buffers let nvrtcGetProgramLog overwrite the shared b" " object #3495

Open

Description

@0z5a

A few places in the repo allocate the NVRTC log buffer as log = b" " * logSize and pass it to nvrtcGetProgramLog. When a compile succeeds with an empty log, logSize is 1. b" " * 1 does not make a new object: it returns CPython's cached one-byte b" ", and NVRTC then writes its terminating NUL into that shared object. After that, every b" " in the process is b"\x00", including the literal, " ".encode() and bytes([32]), and b"a b".split(b" ") no longer splits.

These read the log after every compile, including successful ones:

  • benchmarks/cuda_bindings/runner/runtime.py, compile_cubin()
  • cuda_bindings_13/tests/legacy_api/test_legacy_kernelParams.py, common_nvrtc()

The same line is also in cuda_bindings_12/tests/test_kernelParams.py (common_nvrtc()) and cuda_bindings_12/examples/extra/jit_program_test.py (main()). I have not run those two against cuda-bindings 12.9, so I left them out of the change below; I can add them if you want all four fixed together.

Not affected: cuda_bindings_12/examples/common/common.py only reads the log after a failed compile, where the log is longer than one byte. The cuda.core helpers in _utils/cuda_utils.pyx only allocate when logsize > 1. The b" " * n PTX/CUBIN and LTO-IR data buffers are never one byte long, so they always get a fresh object.

How I found it: I started from NVIDIA-NeMo/Speech#15790, which reports this pattern in NeMo's run_nvrtc(), and searched this repo for b" " * buffers handed to nvrtcGetProgramLog. The places above are the ones that read the log unconditionally.

Repro: I loaded each file unmodified (from main at 0f4983b; the lines are the same on current main), called the function with a trivial kernel, extern "C" __global__ void k(int *p) { if (p) *p = 1; }, and printed the state of b" " before and after the call. nvrtcGetProgramLogSize returned 1 in every run.

file b" " after b"a b".split(b" ") after
benchmarks/cuda_bindings/runner/runtime.py (compile_and_load()) b'\x00' [b'a b']
cuda_bindings_13/tests/legacy_api/test_legacy_kernelParams.py (common_nvrtc()) b'\x00' [b'a b']
same two files with bytearray(logSize) b' ' [b'a', b'b']

" ".encode() and bytes([32]) returned b'\x00' too after the unmodified calls.

Expected: b" " stays b" ", and b"a b".split(b" ") returns [b'a', b'b'].

Environment: Linux aarch64 with an H100-class GPU, driver 595.58.03, Python 3.12.3, cuda-bindings 13.5.0, cuda-pathfinder 1.9.0, numpy 2.5.5, NVRTC 13.2 from the CUDA 13.2 toolkit.

Where this can show up in practice (my estimate; I have not seen a field report):

  • common_nvrtc() is the kind of helper people copy into their own projects. In a copied version, whatever else that process does with b" " afterwards breaks silently: splitting or joining byte strings, padding, building protocol messages. Nothing raises; the results are just wrong.
  • In the test suite, every test that runs after a kernelParams test in the same pytest process sees the corrupted b" ". A failure would then show up in an unrelated test, or a check that compares against b" " could pass when it should not.
  • The benchmark runner compiles every benchmark kernel in one process, so anything later in a run that formats or parses bytes with b" " is affected.

Fix: allocate the buffer with bytearray(logSize). nvrtcGetProgramLog accepts a bytearray, and log.decode() works the same. With that change b" " is unchanged in both runs above.

CONTRIBUTING asks contributors to coordinate before implementing, so I have not opened a PR. The two-line change is on a branch in my fork: 0z5a@3bc7c6d (branch nvrtc-log-bytearray). If this direction looks right, could you assign this to me? I'll then open the PR.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    triageNeeds the team's attention

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions