Skip to content

feat(core): grow KV storage over a stable CUDA virtual address - #195

Draft
kunkunblueberry wants to merge 1 commit into
InferMatrixLab:mainfrom
kunkunblueberry:kunkun/rfc192-vmm-kv-storage
Draft

kunkunblueberry wants to merge 1 commit into
InferMatrixLab:mainfrom
kunkunblueberry:kunkun/rfc192-vmm-kv-storage

Conversation

@kunkunblueberry

Copy link
Copy Markdown
Contributor

Required human contributor answers

TODO(contributor): answer question 1 (outcome) — reserved for the human contributor.

1. What outcome are you trying to achieve?

TODO(contributor)

2. What is the highest-leverage variable for that outcome?

TODO(contributor)

3. How will this work move that variable rather than create busywork?

TODO(contributor)

4. Who is the customer or user, and what do they actually need?

TODO(contributor)

PR necessity and bugfix evidence

TODO(contributor): state whether a PR is needed and why.

Purpose

RFC #192 stage 2: make KV storage physical capacity grow inside one stable CUDA
virtual address range, so the K/V tensors never move.

plan_cache sizes the pool before ModelRunner exists, so any later growth by
reallocation would invalidate pointers already baked into captured CUDA graphs.
This change separates "reserve" from "commit":

  • vllm_rlt/core/vmm_pool.py — GPUKVStorage reserves one VMM range covering
    both planes and maps physical memory into it. commit_blocks() only appends;
    close() is the only release path, so the pool has a single lifetime instead
    of one per tensor.
  • KVCacheManager can take that pool (storage=) and expose
    max_blocks / committed_blocks / published_blocks with publish_capacity().
    published <= committed <= max always holds, and a block is only handed to a
    request after its bytes are mapped, so no allocation can address unmapped
    memory. Without storage= nothing changes.
  • StageEngine maps a bootstrap prefix, builds the cache against it, and maps
    the planned remainder before serving starts. memory_plan records the pool
    accounting.
  • CacheConfig.vmm_bootstrap_blocks / --kv-vmm-bootstrap-blocks gate the
    feature: automatic planning only, CUDA only, rejected with a PD connector.

Scope boundaries: capacity is still plan_cache's estimate (the measured
planner is RFC #192 stage 1), growth happens once during startup, PD/NIXL KV
re-registration is not implemented, and request-level block semantics, prefix
caching and scheduler behaviour are untouched.

Docs: docs/kv_storage.md.

Test Plan

vllm-rlt Version / Commit: 91e3383 (this PR head, based on 13b08ca)

Run on the PR head:

  • python -m ruff check . and python -m ruff format --check . — the same
    commands as cpu-tests.yml.
  • python -m pytest -q (CPU suite) in a worktree of this head.
  • GPU host (A100-SXM4-80GB, shared, CUDA_VISIBLE_DEVICES=1):
    • python -m pytest tests/test_vmm_pool.py --run-gpu -q
    • OURO_CHECKPOINT=<Ouro-1.4B snapshot> python -m pytest tests/test_vmm_real_checkpoint.py --run-gpu -q
    • python -m pytest tests/test_kv_cache.py tests/test_prepared_kv.py tests/test_cuda_graph.py tests/test_stage_engine.py tests/test_prefix_growth.py tests/test_cdb_runtime.py --run-gpu -q
  • Production path with real weights: python -m vllm_rlt.entrypoints.cli twice
    (with and without --kv-vmm-bootstrap-blocks 1) on the Ouro-1.4B snapshot.

Not run: flash_attn_4 paths (unavailable on SM80), PD/NIXL with VMM (rejected
by design in this stage), runtime (post-startup) growth (not implemented).

Test Result

  • Ruff lint and format on the whole tree: pass (141 files already formatted).
  • CPU suite on this head: 1095 passed, 32 failed. A clean checkout of
    13b08ca fails the same 32 tests on the same host
    (test_async_speculative_sampling_plan.py 12, test_serving.py 18,
    test_pd.py 1, test_ouro.py 1), so this PR adds no CPU failures. Values
    were not adjusted to make them pass.
  • GPU storage checks: 11 passed — pointer/shape/stride stability across
    growth, data survival, new-block access, graph replay after growth, release
    accounting (2 GiB mapping), publish/commit bound, engine-level parity on the
    torch and Triton backends.
  • GPU real-checkpoint checks (ByteDance/Ouro-1.4B snapshot, BF16): 5 passed
    — identical generated tokens and identical KV bytes (three layers, same
    positions) versus the plain torch.empty pool, CUDA graph capture with
    Triton, prefix-cache hits equal to the baseline, a clamped bootstrap, and
    repeated create/close leaving no mapped bytes or tensors behind.
  • Regression suites above: 217 passed, 28 failed; every failure also fails on
    a clean 13b08ca on the same host (flash_attn_4 unsupported on SM80,
    ExecutionConfig equality, allocator-residue, one ModuleNotFoundError in
    test_cdb_runtime.py).
  • Production CLI with real weights: token_ids, exit_depths and text
    identical with and without --kv-vmm-bootstrap-blocks 1
    ([7042, 30, 198, 198, 504, 3575, 282, 4649], " Paris.\n\nThe capital of France").
  • Two defects were found and fixed during this validation: the allocator's
    cached profile-probe segment blocked cuMemCreate (the pool now returns it
    first), and bytes_per_block already counts K and V, so the pool must map
    half of it per plane (it previously over-committed by 2x).

Reserve one CUDA VMM range for the K and V planes, map only a bootstrap
prefix at startup, and map the remainder before serving starts. The K/V
tensors keep their pointer, shape and stride across growth, so a graph
captured against them stays valid while the pool grows.

- GPUKVStorage owns the reservation, the mappings and their lifetime, and
  KVCacheManager publishes capacity from it instead of reallocating tensors.
- published <= committed <= max is enforced, and a block is only handed out
  once its bytes are mapped, so no allocation can reach unmapped memory.
- CacheConfig.vmm_bootstrap_blocks / --kv-vmm-bootstrap-blocks enable it for
  automatic planning on CUDA without a PD connector; growth happens once,
  during startup.
- Capacity is still plan_cache's estimate: this is RFC InferMatrixLab#192 stage 2, not the
  measured planner.

Verified on an A100: 11 storage checks plus 5 real-checkpoint checks against
the Ouro-1.4B snapshot (generated tokens, KV bytes in three layers, CUDA graph
capture, prefix-cache hits, repeated engine create/close).

Signed-off-by: kunkunblueberry <kunkunblueberry@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant