Repository navigation
feat(kv): add CUDA VMM KV storage with fixed addresses - #194
XinhaoTheo wants to merge 1 commit into
Conversation
Add StableKVStorage for RFC InferMatrixLab#192. It reserves one virtual address range for each KV pool. The tensor of each pool has the maximum block count from the start. commit() adds physical memory to more blocks. It does not change a tensor address, a stride or a stored value. Because of this, a CUDA Graph that was captured before a commit also writes correctly to the new blocks. The driver binding uses ctypes and libcuda.so.1. It does not need a build step or a new dependency. vmm_unavailable_reason() gives the reason when a device cannot use this storage. The engine does not use this storage yet.
kunkunblueberry
left a comment
There was a problem hiding this comment.
Thanks — this is a useful piece. I read the diff and ran your two files on a clean upstream/main checkout on an A100 (torch 2.14+cu130), so the observations below are reproduced, not inferred. I am the author of #195, the end-to-end variant of the same RFC #192 step, so the overlap is discussed at the end.
Reviewed head: 7aaa7c6 (base c338aaa, 11 commits behind main).
Blocking
1. commit() has no upper bound — vllm_rlt/core/kv_storage.py:320
The docstring says blocks must not exceed max_blocks, but nothing enforces it. With 3 MiB blocks and max_blocks=1 (4 MiB reserved), commit(2) asks for 6 MiB and fails inside the driver:
commit(2) raised: RuntimeError : CUDA driver error 1: invalid argument
An integration mistake would surface as an opaque driver error during startup. if not 1 <= blocks <= self.max_blocks: raise ValueError(...) turns it into a usable message. Note that when the block is smaller than the granule the round-up hides it: max_blocks=4 with 64 KiB blocks, commit(5) silently returns 4.
2. close() leaves the object looking alive — :336
After close() on a 64-block / 2 MiB store:
before close: committed_blocks = 64 committed_bytes = 2097152
after close: committed_blocks = 64 committed_bytes = 2097152
after close: tensors still exposed = 1
commit after close -> 64
write through a stale view after close: no error
The VA has already been freed at that point, so this is a silent-corruption window rather than a crash: commit() is a no-op that reports success, and a surviving view still writes. A _closed flag that raises from commit() and from tensors, plus zeroed counters, makes the misuse loud.
3. ruff format --check . fails, which is what cpu-tests.yml runs (pinned ruff==0.13.2)
Would reformat: tests\test_kv_storage.py
Would reformat: vllm_rlt\core\kv_storage.py
Reproduced with 0.13.2 itself, so it is not a version artifact. ruff check . passes.
Non-blocking
4. initial_blocks=0 fails deep inside torch — :308: RuntimeError: The specified pointer resides on host memory and is not registered with any CUDA device. The constraint comes from building the tensors out of the pool before any mapping exists, not from CUDA, so an explicit validation would be clearer.
5. No capacity gate for the consumer. tensors spans max_blocks, and "do not access the blocks after committed_blocks" is documentation only. Once KVCacheManager is wired (your unchecked item), allocating past committed_blocks is a sticky illegal address that poisons the CUDA context for the whole process. A published <= committed <= max split, where publishing requires committed_blocks, makes that unreachable; your TODO already lists the free-pool update, and the invariant is what keeps it safe.
6. No CI evidence attached — statusCheckRollup is empty and mergeStateStatus is UNSTABLE, so the H200/H100 runs are not visible from the PR. Rebasing onto current main would also help, since the base is 11 commits behind.
7. Test oracle depends on absolute driver memory — test_close_returns_memory_the_torch_allocator_never_counted compares mem_get_info() values. That is fine on a dedicated Modal box, but another tenant's allocations move the same counter, so it flakes on a shared device (I hit exactly this in my own suite). A sizeable mapping plus release accounting from the storage itself is more stable.
Verified as correct, for the record
- The lifetime story holds:
torch.as_tensorkeeps the__cuda_array_interface__provider alive, so views keep the pool alive. Reproduced: afterdel storagethe finalizer did not run and the surviving view stayed writable. weakref.finalizereceives a module-level function rather than a bound method, which avoids the self-reference keepalive trap.close()is idempotent, a failedcuMemMapreleases its handle, andmemory_reserved()stays untouched, so "the PyTorch allocator does not count this memory" is accurate.- Driving the real
write_graph_kvkernel in the graph test is a stronger check than a torch-only write.
Overlap with #195
#195 implements the same RFC #192 step end-to-end (storage + KVCacheManager capacity states + startup growth + CLI flag) and also adds tests/test_kv_storage.py, so the two will conflict. Two storage modules for one mechanism is the part worth avoiding. My suggestion:
- keep a single storage layer — your
PoolSpecAPI is more general (it can host the #147/#174 layouts) and yourvmm_unavailable_reasonprobe is better than mine; - port items 1, 2 and 5 into it;
- then I rebase #195 onto your module, drop my
vmm_pool.pyand rename my test file.
I am happy to do that rebase once 1-3 are addressed, if you prefer to keep the storage module where it is.
Required human contributor answers
1. What outcome are you trying to achieve?
Size the KV pool from the memory that is free after warmup and CUDA Graph capture (RFC #192). The captured graphs must stay valid.
2. What is the highest-leverage variable for that outcome?
The KV tensor addresses. A captured graph keeps the KV pointers, the strides and the size of the first dimension (
cuda_graph.py#L121). If the addresses stay fixed, the engine can add KV capacity after capture.3. How will this work move that variable rather than create busywork?
StableKVStoragereserves the full virtual address range at startup. It adds physical memory later with CUDA VMM. The addresses, strides and stored values do not change. The tests show that a graph that was captured before a commit writes correctly to the new blocks.4. Who is the customer or user, and what do they actually need?
PR necessity
A PR is necessary. No current storage path keeps the KV addresses fixed when the capacity increases. This is not a bugfix.
Modification
This draft adds
vllm_rlt/core/kv_storage.py. The engine does not use it yet, so no behavior changes.PoolSpec(block_shape, dtype)gives the shape and dtype of one block in one pool. A pool is not specific to K or V, so other page layouts ([KV] Pack recurrent depth planes for partially recurrent models #147, feat(kv): store INT4/NVFP4 KV pages in the cache manager and read them with Triton #174) can add pools.StableKVStorage(specs, max_blocks=..., initial_blocks=..., device=...):[max_blocks, *block_shape].commit(blocks)adds physical memory from the start of each range, in multiples of the granule (2 MiB on H100). The capacity can only increase.committed_bytesgives the physical bytes. The PyTorch allocator does not count them.close()synchronizes the device, then releases all memory. The memory is also released when no reference to the storage or its tensor views remains.vmm_unavailable_reason(device)gives the reason for a fallback: not a CUDA device, a ROCm build, not Linux, or no VMM support.ctypesandlibcuda.so.1. It does not need a build step or a new dependency.Out of scope: the startup order change and the measured final capacity (other RFC #192 contributors), HIP VMM for ROCm, and VMM memory for NIXL RDMA.
Related PR
commit()andcommitted_bytes.kv_cache_manager.py. The next commit changes the same code.torch.emptyfallback.enable_extensible_kv_cacheinvllm/config/cache.pyandvllm/utils/vmm_driver.py.Accuracy Test
This commit does not change an execution path, so the model outputs do not change.
The new tests in
tests/test_kv_storage.pycheck three things:commit().close()releases the memory.All tests passed on H200 and H100 (Modal). The CPU regression suite also passed. The token equality test against
torch.emptycomes with the engine integration.Benchmark and Profiling
Micro-benchmark on one H100 80GB HBM3. The KV geometry comes from the
ByteDance/Ouro-1.4Bconfig.json: 24 layers, 16 KV heads, head dim 128, block size 16, BF16. One block of one pool is 1.5 MiB.Lifecycle cost (K and V pools together, median of 3 runs):
commit()close()torch.emptyallocatetorch.emptyfreeRounding: the granule is 2 MiB. The unused physical memory is 1 MiB to 2 MiB for K and V together. The usable block count is equal to the requested block count.
Bandwidth (both storages in memory, 6 interleaved rounds, median):
torch.emptypaged_attention, batch 32, context 4096, 24 layersThe attention outputs of the two storages are bitwise equal.
Profiling: not applicable, because no execution path changes. The end-to-end A/B and the profiles come with the engine integration.
Conclusion
The KV storage is complete and tested. The engine integration is the next step in this PR.
StableKVStorage: fixed addresses,commit(),close()and the fallback reason.KVCacheManagertoStableKVStorage. Usetorch.emptyand record the reason when VMM is not available, or when a PD/NIXL connector is active.memory_plan. Release the storage inStageEngine.close().torch.empty: Ouro, Huginn and Dory; eager and CUDA Graph; synchronous and asynchronous; speculative decoding.