Skip to content

feat(kv): add CUDA VMM KV storage with fixed addresses - #194

Draft
XinhaoTheo wants to merge 1 commit into
InferMatrixLab:mainfrom
XinhaoTheo:feat/rfc192-vmm-kv-storage
Draft

XinhaoTheo wants to merge 1 commit into
InferMatrixLab:mainfrom
XinhaoTheo:feat/rfc192-vmm-kv-storage

Conversation

@XinhaoTheo

Copy link
Copy Markdown

Required human contributor answers

1. What outcome are you trying to achieve?

Size the KV pool from the memory that is free after warmup and CUDA Graph capture (RFC #192). The captured graphs must stay valid.

2. What is the highest-leverage variable for that outcome?

The KV tensor addresses. A captured graph keeps the KV pointers, the strides and the size of the first dimension (cuda_graph.py#L121). If the addresses stay fixed, the engine can add KV capacity after capture.

3. How will this work move that variable rather than create busywork?

StableKVStorage reserves the full virtual address range at startup. It adds physical memory later with CUDA VMM. The addresses, strides and stored values do not change. The tests show that a graph that was captured before a commit writes correctly to the new blocks.

4. Who is the customer or user, and what do they actually need?

PR necessity

A PR is necessary. No current storage path keeps the KV addresses fixed when the capacity increases. This is not a bugfix.

Modification

This draft adds vllm_rlt/core/kv_storage.py. The engine does not use it yet, so no behavior changes.

  • PoolSpec(block_shape, dtype) gives the shape and dtype of one block in one pool. A pool is not specific to K or V, so other page layouts ([KV] Pack recurrent depth planes for partially recurrent models #147, feat(kv): store INT4/NVFP4 KV pages in the cache manager and read them with Triton #174) can add pools.
  • StableKVStorage(specs, max_blocks=..., initial_blocks=..., device=...):
    • It reserves one virtual address range for each pool. Each pool has one tensor with the shape [max_blocks, *block_shape].
    • commit(blocks) adds physical memory from the start of each range, in multiples of the granule (2 MiB on H100). The capacity can only increase.
    • committed_bytes gives the physical bytes. The PyTorch allocator does not count them.
    • close() synchronizes the device, then releases all memory. The memory is also released when no reference to the storage or its tensor views remains.
  • vmm_unavailable_reason(device) gives the reason for a fallback: not a CUDA device, a ROCm build, not Linux, or no VMM support.
  • The driver binding uses ctypes and libcuda.so.1. It does not need a build step or a new dependency.

Out of scope: the startup order change and the measured final capacity (other RFC #192 contributors), HIP VMM for ROCm, and VMM memory for NIXL RDMA.

Related PR

Accuracy Test

This commit does not change an execution path, so the model outputs do not change.

The new tests in tests/test_kv_storage.py check three things:

  • The addresses stay fixed after commit().
  • A graph that was captured before a commit writes correctly to the new blocks.
  • close() releases the memory.

All tests passed on H200 and H100 (Modal). The CPU regression suite also passed. The token equality test against torch.empty comes with the engine integration.

Benchmark and Profiling

Micro-benchmark on one H100 80GB HBM3. The KV geometry comes from the ByteDance/Ouro-1.4B config.json: 24 layers, 16 KV heads, head dim 128, block size 16, BF16. One block of one pool is 1.5 MiB.

Lifecycle cost (K and V pools together, median of 3 runs):

Size of each pool VMM construct VMM commit() VMM close() torch.empty allocate torch.empty free
1 GiB 2.7 ms 2.0 ms 1.3 ms 1.8 ms 1.3 ms
8 GiB 2.4 ms 3.2 ms 8.8 ms 5.2 ms 7.6 ms
32 GiB 2.6 ms 143 ms 37 ms 180 ms 41 ms

Rounding: the granule is 2 MiB. The unused physical memory is 1 MiB to 2 MiB for K and V together. The usable block count is equal to the requested block count.

Bandwidth (both storages in memory, 6 interleaved rounds, median):

Workload VMM torch.empty Difference
Triton paged_attention, batch 32, context 4096, 24 layers 2932.9 GB/s 2933.3 GB/s −0.02% (noise)
Device copy, 12 GiB 3033.0 GB/s 3033.9 GB/s −0.03% (all rounds)

The attention outputs of the two storages are bitwise equal.

Profiling: not applicable, because no execution path changes. The end-to-end A/B and the profiles come with the engine integration.

Conclusion

The KV storage is complete and tested. The engine integration is the next step in this PR.

  • Add StableKVStorage: fixed addresses, commit(), close() and the fallback reason.
  • Test it on GPU: fixed addresses, graph replay on new blocks, and memory release.
  • Measure the lifecycle cost, the rounding and the bandwidth on H100.
  • Connect KVCacheManager to StableKVStorage. Use torch.empty and record the reason when VMM is not available, or when a PD/NIXL connector is active.
  • Add an operation that adds newly committed blocks to the free block pool.
  • Record the storage type, the reserved bytes and the committed bytes in memory_plan. Release the storage in StageEngine.close().
  • Test greedy token equality against torch.empty: Ouro, Huginn and Dory; eager and CUDA Graph; synchronous and asynchronous; speculative decoding.
  • Run the end-to-end A/B on H100 without the profiler, then the ops-only and full profiles.

Add StableKVStorage for RFC InferMatrixLab#192. It reserves one virtual address range
for each KV pool. The tensor of each pool has the maximum block count from
the start. commit() adds physical memory to more blocks. It does not change
a tensor address, a stride or a stored value. Because of this, a CUDA Graph
that was captured before a commit also writes correctly to the new blocks.

The driver binding uses ctypes and libcuda.so.1. It does not need a build
step or a new dependency. vmm_unavailable_reason() gives the reason when a
device cannot use this storage.

The engine does not use this storage yet.

@kunkunblueberry kunkunblueberry left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — this is a useful piece. I read the diff and ran your two files on a clean upstream/main checkout on an A100 (torch 2.14+cu130), so the observations below are reproduced, not inferred. I am the author of #195, the end-to-end variant of the same RFC #192 step, so the overlap is discussed at the end.

Reviewed head: 7aaa7c6 (base c338aaa, 11 commits behind main).

Blocking

1. commit() has no upper bound — vllm_rlt/core/kv_storage.py:320

The docstring says blocks must not exceed max_blocks, but nothing enforces it. With 3 MiB blocks and max_blocks=1 (4 MiB reserved), commit(2) asks for 6 MiB and fails inside the driver:

commit(2) raised: RuntimeError : CUDA driver error 1: invalid argument

An integration mistake would surface as an opaque driver error during startup. if not 1 <= blocks <= self.max_blocks: raise ValueError(...) turns it into a usable message. Note that when the block is smaller than the granule the round-up hides it: max_blocks=4 with 64 KiB blocks, commit(5) silently returns 4.

2. close() leaves the object looking alive — :336

After close() on a 64-block / 2 MiB store:

before close: committed_blocks = 64 committed_bytes = 2097152
after  close: committed_blocks = 64 committed_bytes = 2097152
after  close: tensors still exposed = 1
commit after close -> 64
write through a stale view after close: no error

The VA has already been freed at that point, so this is a silent-corruption window rather than a crash: commit() is a no-op that reports success, and a surviving view still writes. A _closed flag that raises from commit() and from tensors, plus zeroed counters, makes the misuse loud.

3. ruff format --check . fails, which is what cpu-tests.yml runs (pinned ruff==0.13.2)

Would reformat: tests\test_kv_storage.py
Would reformat: vllm_rlt\core\kv_storage.py

Reproduced with 0.13.2 itself, so it is not a version artifact. ruff check . passes.

Non-blocking

4. initial_blocks=0 fails deep inside torch — :308: RuntimeError: The specified pointer resides on host memory and is not registered with any CUDA device. The constraint comes from building the tensors out of the pool before any mapping exists, not from CUDA, so an explicit validation would be clearer.

5. No capacity gate for the consumer. tensors spans max_blocks, and "do not access the blocks after committed_blocks" is documentation only. Once KVCacheManager is wired (your unchecked item), allocating past committed_blocks is a sticky illegal address that poisons the CUDA context for the whole process. A published <= committed <= max split, where publishing requires committed_blocks, makes that unreachable; your TODO already lists the free-pool update, and the invariant is what keeps it safe.

6. No CI evidence attached — statusCheckRollup is empty and mergeStateStatus is UNSTABLE, so the H200/H100 runs are not visible from the PR. Rebasing onto current main would also help, since the base is 11 commits behind.

7. Test oracle depends on absolute driver memory — test_close_returns_memory_the_torch_allocator_never_counted compares mem_get_info() values. That is fine on a dedicated Modal box, but another tenant's allocations move the same counter, so it flakes on a shared device (I hit exactly this in my own suite). A sizeable mapping plus release accounting from the storage itself is more stable.

Verified as correct, for the record

  • The lifetime story holds: torch.as_tensor keeps the __cuda_array_interface__ provider alive, so views keep the pool alive. Reproduced: after del storage the finalizer did not run and the surviving view stayed writable.
  • weakref.finalize receives a module-level function rather than a bound method, which avoids the self-reference keepalive trap.
  • close() is idempotent, a failed cuMemMap releases its handle, and memory_reserved() stays untouched, so "the PyTorch allocator does not count this memory" is accurate.
  • Driving the real write_graph_kv kernel in the graph test is a stronger check than a torch-only write.

Overlap with #195

#195 implements the same RFC #192 step end-to-end (storage + KVCacheManager capacity states + startup growth + CLI flag) and also adds tests/test_kv_storage.py, so the two will conflict. Two storage modules for one mechanism is the part worth avoiding. My suggestion:

  • keep a single storage layer — your PoolSpec API is more general (it can host the #147/#174 layouts) and your vmm_unavailable_reason probe is better than mine;
  • port items 1, 2 and 5 into it;
  • then I rebase #195 onto your module, drop my vmm_pool.py and rename my test file.

I am happy to do that rebase once 1-3 are addressed, if you prefer to keep the storage module where it is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants