Repository navigation
feat(core): grow KV storage over a stable CUDA virtual address - #195
Draft
kunkunblueberry wants to merge 1 commit into
Draft
kunkunblueberry wants to merge 1 commit into
kunkunblueberry wants to merge 1 commit into
Conversation
Reserve one CUDA VMM range for the K and V planes, map only a bootstrap prefix at startup, and map the remainder before serving starts. The K/V tensors keep their pointer, shape and stride across growth, so a graph captured against them stays valid while the pool grows. - GPUKVStorage owns the reservation, the mappings and their lifetime, and KVCacheManager publishes capacity from it instead of reallocating tensors. - published <= committed <= max is enforced, and a block is only handed out once its bytes are mapped, so no allocation can reach unmapped memory. - CacheConfig.vmm_bootstrap_blocks / --kv-vmm-bootstrap-blocks enable it for automatic planning on CUDA without a PD connector; growth happens once, during startup. - Capacity is still plan_cache's estimate: this is RFC InferMatrixLab#192 stage 2, not the measured planner. Verified on an A100: 11 storage checks plus 5 real-checkpoint checks against the Ouro-1.4B snapshot (generated tokens, KV bytes in three layers, CUDA graph capture, prefix-cache hits, repeated engine create/close). Signed-off-by: kunkunblueberry <kunkunblueberry@gmail.com>
3 of 8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Required human contributor answers
TODO(contributor): answer question 1 (outcome) — reserved for the human contributor.
1. What outcome are you trying to achieve?
TODO(contributor)
2. What is the highest-leverage variable for that outcome?
TODO(contributor)
3. How will this work move that variable rather than create busywork?
TODO(contributor)
4. Who is the customer or user, and what do they actually need?
TODO(contributor)
PR necessity and bugfix evidence
TODO(contributor): state whether a PR is needed and why.
Purpose
RFC #192 stage 2: make KV storage physical capacity grow inside one stable CUDA
virtual address range, so the K/V tensors never move.
plan_cachesizes the pool beforeModelRunnerexists, so any later growth byreallocation would invalidate pointers already baked into captured CUDA graphs.
This change separates "reserve" from "commit":
vllm_rlt/core/vmm_pool.py—GPUKVStoragereserves one VMM range coveringboth planes and maps physical memory into it.
commit_blocks()only appends;close()is the only release path, so the pool has a single lifetime insteadof one per tensor.
KVCacheManagercan take that pool (storage=) and exposemax_blocks / committed_blocks / published_blockswithpublish_capacity().published <= committed <= maxalways holds, and a block is only handed to arequest after its bytes are mapped, so no allocation can address unmapped
memory. Without
storage=nothing changes.StageEnginemaps a bootstrap prefix, builds the cache against it, and mapsthe planned remainder before serving starts.
memory_planrecords the poolaccounting.
CacheConfig.vmm_bootstrap_blocks/--kv-vmm-bootstrap-blocksgate thefeature: automatic planning only, CUDA only, rejected with a PD connector.
Scope boundaries: capacity is still
plan_cache's estimate (the measuredplanner is RFC #192 stage 1), growth happens once during startup, PD/NIXL KV
re-registration is not implemented, and request-level block semantics, prefix
caching and scheduler behaviour are untouched.
Docs:
docs/kv_storage.md.Test Plan
vllm-rlt Version / Commit:
91e3383(this PR head, based on13b08ca)Run on the PR head:
python -m ruff check .andpython -m ruff format --check .— the samecommands as
cpu-tests.yml.python -m pytest -q(CPU suite) in a worktree of this head.CUDA_VISIBLE_DEVICES=1):python -m pytest tests/test_vmm_pool.py --run-gpu -qOURO_CHECKPOINT=<Ouro-1.4B snapshot> python -m pytest tests/test_vmm_real_checkpoint.py --run-gpu -qpython -m pytest tests/test_kv_cache.py tests/test_prepared_kv.py tests/test_cuda_graph.py tests/test_stage_engine.py tests/test_prefix_growth.py tests/test_cdb_runtime.py --run-gpu -qpython -m vllm_rlt.entrypoints.clitwice(with and without
--kv-vmm-bootstrap-blocks 1) on the Ouro-1.4B snapshot.Not run:
flash_attn_4paths (unavailable on SM80), PD/NIXL with VMM (rejectedby design in this stage), runtime (post-startup) growth (not implemented).
Test Result
141 files already formatted).1095 passed, 32 failed. A clean checkout of13b08cafails the same 32 tests on the same host(
test_async_speculative_sampling_plan.py12,test_serving.py18,test_pd.py1,test_ouro.py1), so this PR adds no CPU failures. Valueswere not adjusted to make them pass.
11 passed— pointer/shape/stride stability acrossgrowth, data survival, new-block access, graph replay after growth, release
accounting (2 GiB mapping), publish/commit bound, engine-level parity on the
torch and Triton backends.
ByteDance/Ouro-1.4Bsnapshot, BF16):5 passed— identical generated tokens and identical KV bytes (three layers, same
positions) versus the plain
torch.emptypool, CUDA graph capture withTriton, prefix-cache hits equal to the baseline, a clamped bootstrap, and
repeated create/close leaving no mapped bytes or tensors behind.
217 passed, 28 failed; every failure also fails ona clean
13b08caon the same host (flash_attn_4unsupported on SM80,ExecutionConfigequality, allocator-residue, oneModuleNotFoundErrorintest_cdb_runtime.py).token_ids,exit_depthsandtextidentical with and without
--kv-vmm-bootstrap-blocks 1(
[7042, 30, 198, 198, 504, 3575, 282, 4649]," Paris.\n\nThe capital of France").cached profile-probe segment blocked
cuMemCreate(the pool now returns itfirst), and
bytes_per_blockalready counts K and V, so the pool must maphalf of it per plane (it previously over-committed by 2x).