Repository navigation
Description
Motivation
Size the KV pool from the memory available after actual runtime initialization, warmup and CUDA Graph capture, while keeping KV addresses stable. Replace fixed per-graph-pool allowances with measured usage and an explicit allowance for runtime transients.
At the inspected baseline, plan_cache() runs a temporary prelude/recurrent/coda forward, measures its peak, and subtracts estimated execution buffers, metadata and graph headroom. StageEngine then allocates the final KV tensors before constructing ModelRunner. Graph budgeting includes a fixed allowance per pool and a separate estimate for retained speculative logits.
Warmup already exists: _capture() executes the path twice before capture. Some asynchronous speculative graphs are prepared at startup; ordinary graph entries are mainly captured lazily. What is missing is a coordinated startup lifecycle that completes the necessary work before final KV sizing.
This is a design proposal based on source inspection; no capacity or performance improvement has been measured.
Proposed change
1. Separate three allocation decisions
- Storage capacity: how much physical GPU memory backs the engine-wide KV pool.
- Request ownership: which committed KV blocks a request currently owns; allocate these incrementally as needed.
- Logical addressing: metadata maps layer, loop and token identity to stored KV.
This RFC changes startup profiling and storage capacity. It does not require reserving all future output blocks for each request or organizing physical storage into preallocated depth planes. It should work with the independently discussed KV page-layout changes in #147.
GPU virtual-memory pages and token blocks are different units. Mapping alignment must be handled by storage; the request allocator only exposes fully backed, usable KV blocks.
2. Change startup order
load model and select backend
-> reserve a stable virtual address range for KV
-> commit enough physical storage for valid warmup inputs
-> initialize the actual runner, buffers, metadata and enabled connectors
-> warm the configured execution paths
-> capture the planned graph entries and exercise replay
-> measure remaining runtime transient requirements
-> drain all startup work and release temporary requests/tensors
-> determine and commit final KV capacity
-> publish the usable block pool and accept requests
The virtual reservation has a bounded upper limit derived from supported capacity and device constraints. It does not consume that amount of physical VRAM.
Warmup uses scratch requests whose accessed KV addresses are mapped and valid. Masked/inactive rows must never access unmapped storage. Scratch state must not enter prefix caches, consume real-request RNG state, or survive into serving.
After final sizing, verify replay and a representative execution within the measured budget before declaring the engine ready.
3. Preserve captured addresses
Replace direct allocation of the final K/V tensors with storage that can add physical mappings inside a stable virtual range.
- Captured pointers, tensor views and strides remain valid across physical commitment.
- Runtime metadata controls valid block indices; reserved virtual capacity is never treated as committed capacity.
- The logical allocator receives only committed blocks after sizing.
- One storage owner manages mappings and tensor lifetimes. Unmap/free only after dependent device work and graph users have retired.
- Account for these allocations explicitly: driver/VMM allocations may not appear in PyTorch allocator statistics.
This requires a GPU storage implementation, not simply reallocating or resizing a PyTorch tensor after capture. GPU VMM operations belong outside the request execution hot path.
4. Warm the paths enabled by the configuration
Warmup should trigger compilation, CUDA library initialization and actual workspace allocation, and establish a representative memory envelope. It is not a full benchmark suite.
Cover the relevant prefill token limits, decode batches and graph buckets, boundary-KV stages, packed/staged prefill, and speculative verification when enabled. Include context lengths that affect attention scratch usage. A single largest batch is insufficient when shapes select different kernels or workspaces.
For asynchronous/multi-stream execution, exercise supported overlap and buffer lifetimes. Measure the concurrent envelope rather than assuming it is twice a serial forward peak.
Use existing per-loop execution and scheduling boundaries. Warmup and capture must preserve mixed admission semantics.
5. Measure persistent usage and incremental transient demand
Record driver-free memory alongside PyTorch allocated/reserved/peak statistics at phase boundaries. Drain participating streams before each boundary measurement and apply a consistent temporary-object/cache cleanup policy.
Distinguish:
- Persistent: weights, execution buffers, metadata, captured graphs and retained outputs.
- Transient: additional physical memory needed during subsequent execution above that persistent baseline.
- Bootstrap KV: mapped scratch storage, tracked separately from model workspace.
After cleanup, persistent allocations already reduce driver-free memory. Do not subtract graph memory again, or count temporary probe KV as runtime workspace. PyTorch caching and reusable workspace must be accounted for consistently so that the same bytes are not reserved twice.
Final KV capacity consists of retained bootstrap mappings plus additional mappings that fit within:
- measured remaining physical memory;
- measured additional runtime transient needs and a safety margin;
- the remaining configured instance-memory budget;
- supported/useful capacity and mapping alignment.
Keep gpu_memory_utilization as a cap; this RFC does not change its default to 1.0. Preserve explicit num_blocks / kv_cache_memory_bytes as capacity overrides with clear startup failure if the requested capacity cannot coexist with execution. Overrides may skip automatic sizing, but must not disable required capture preparation.
Log startup phase costs, graph entry coverage, persistent usage, transient allowance, safety margin, committed KV bytes and usable block count. Do not consume all post-capture free memory: later eager execution still needs working space.
6. Resolve lazy graph growth before removing graph headroom
For the measured path, define a bounded graph-entry plan from the configured limits and complete its capture before final sizing. Unplanned shapes use the existing eager fallback, whose workspace is included in warmup coverage.
Changing graph shape keys or bucket padding requires output/metadata validation, including retained output lifetimes and shared graph-pool replay ordering. A bucketed shape must not alter logical loop scheduling.
If a path still permits serving-time capture, retain an explicit bounded budget for it and report that coverage gap. Do not claim fully measured capacity or remove all graph headroom while future graph allocations remain unaccounted for.
Implementation boundaries and rollout
- Coordinate startup warmup and capture using the existing execution paths; add phase measurements. Retain the current allocation strategy while this is introduced.
- Add stable-address GPU KV storage, integrate final capacity publication, and validate it with capture/replay.
- Enable measured sizing on supported configurations. Retire fixed per-pool graph estimates where capture coverage is complete.
StageEngine coordinates initialization. Execution components own warmup/capture and device buffers; memory planning calculates capacity; storage owns mappings; the logical KV manager owns block leases and request mappings. Avoid a second scheduler or duplicate request-state model.
Select supported behavior automatically. Keep the existing bounded CPU behavior and an identified estimate-based fallback for unsupported VMM/platform/connector combinations. Validate connector registration/export requirements before selecting the storage path; report the fallback reason. Clean up partial mappings and graphs if startup fails.
Alternatives and expected impact
- Keep fixed graph reserves: simple, but continues to rely on estimates that vary with enabled paths and retained outputs.
- Allocate a small ordinary KV tensor, then replace it after capture: invalidates captured pointers; recapturing against a larger tensor reintroduces the capacity/capture dependency.
- Use every byte free after capture: leaves no allowance for later transient execution.
- Commit physical storage per request during serving: unnecessary for this startup proposal; requests can incrementally lease blocks from a committed pool.
Expected tradeoff: more predictable serving startup and potentially better use of VRAM, at the cost of more startup work and GPU storage complexity. The capacity gain is configuration-dependent and may be small or negative where existing estimates were insufficient.
Validation
- Verify captured pointers/views/strides remain valid through commitment and that previously captured graphs replay using newly committed blocks.
- Compare generated tokens/depths and KV contents against the existing path for boundary-KV and recurrent models, eager/graph, supported synchronous/asynchronous paths, mixed scheduling, and enabled speculation.
- After sizing, run representative high-memory prefill, decode and supported overlaps; verify transient headroom is sufficient.
- Exercise partial-startup failure, cleanup, repeated engine creation, block reuse and supported connector lifetimes.
- Compare baseline/candidate startup duration, committed KV capacity, physical peak usage, first-request latency and steady-state performance under matched settings. Report limitations and untested combinations.
Preemption snapshot/recomputation remains tracked separately in #191.
References
- vLLM extensible KV configuration: stable virtual reservation and physical commitment after warmup/capture.
- vLLM memory sizing: measured remaining memory, transient allowance and safety margin.
Activity
kunkunblueberry commentedon Nov 16, 2026
I'd like to contribute to the full runtime KV memory work across the three related PRs. These parts are tightly coupled, so I think it is important to understand and help implement the end-to-end path rather than treat capacity planning or storage initialization in isolation.
In particular, I'm interested in:
I'm happy to work across all three PRs and help keep the interfaces and lifecycle consistent. We can coordinate the concrete split of implementation work as the design develops.
cecilsd commentedon Nov 16, 2026
I'd like to take the startup warmup and CUDA Graph capture orchestration part of step 1. I'll focus on running the existing warmup paths during StageEngine startup, capturing a bounded set of graph entries, and checking replay before the engine accepts requests. The first change will use the current KV allocation strategy.
I'll coordinate with @kunkunblueberry on where warmup and capture hand off to memory measurement. KV sizing, stable-address storage, and final capacity publication are outside my scope.
XinhaoTheo commentedon Nov 16, 2026
I like to take on the VMM storage backend, reserving stable KV addresses and committing physical memory without invalidating captured graphs.
@kunkunblueberry @cecilsd, would that split work for you? I will focus on the storage backend and its lifecycle, coordinating with your capacity planning and warmup/capture work on the interfaces.
bjf-frz commentedon Nov 17, 2026
Thanks all, assigned to @kunkunblueberry, @cecilsd and @XinhaoTheo. The split proposed here works:
Please agree on the hand-off interfaces (warmup/capture → measurement → storage commitment) here before the PRs depend on each other.
XinhaoTheo commentedon Nov 17, 2026
Agree
feat(kv): add CUDA VMM KV storage with fixed addresses
cecilsd commentedon Nov 18, 2026
@kunkunblueberry @XinhaoTheo I've opened PR #198 for startup warmup and CUDA Graph capture orchestration, based on
bed3400, following the agreed division of work. It retains the current KV allocation strategy and has no dependency on the other implementation branches. The following is a proposed handoff for us to agree on before integration.The worker now initializes/registers its connector, calls the idempotent
StageEngine.prepare_runtime(), and sendsreadyonly after preparation succeeds. Preparation uses the actual serving runners and physically valid scratch KV leases. Scratch requests do not enter the scheduler or publish prefix-cache entries. The successful return and cleanedstartup_reportprovide the current step-1 handoff; they do not calculate or approve a new KV capacity.Measurement boundary and ownership
empty_cacheis called between phases (allocator_cache_policy="retain").num_blocksandkv_bytesdescribe the currently allocated pool, including FP8 scales. They are not measured-sizing results. Measurement should account for KV separately and avoid subtracting already-resident Graph memory again.Proposed integration sequence
Storage needs to preserve captured pointers, tensor views, and strides across commitment, and publish only fully backed usable blocks. Failure cleanup must drain dependent device work before releasing mappings or owners. The current PR does not implement VMM, final measured sizing, or post-commit validation; those belong to the subsequent integration.
Validation
pre-commit run --all-filespassed. The full CPU suite recorded 1236 passed / 35 skipped.Does this drained/cleaned handoff and report semantics fit your measurement/storage work? We should also agree on how
StageEnginecoordinates validation after final storage commitment, and on connector registration requirements for the mapping lifecycle, before fixing the cross-PR interfaces.