Skip to content

[RFC]: Size KV storage after runtime warmup and CUDA Graph capture #192

Open

Description

@bjf-frz

Motivation

Size the KV pool from the memory available after actual runtime initialization, warmup and CUDA Graph capture, while keeping KV addresses stable. Replace fixed per-graph-pool allowances with measured usage and an explicit allowance for runtime transients.

At the inspected baseline, plan_cache() runs a temporary prelude/recurrent/coda forward, measures its peak, and subtracts estimated execution buffers, metadata and graph headroom. StageEngine then allocates the final KV tensors before constructing ModelRunner. Graph budgeting includes a fixed allowance per pool and a separate estimate for retained speculative logits.

Warmup already exists: _capture() executes the path twice before capture. Some asynchronous speculative graphs are prepared at startup; ordinary graph entries are mainly captured lazily. What is missing is a coordinated startup lifecycle that completes the necessary work before final KV sizing.

This is a design proposal based on source inspection; no capacity or performance improvement has been measured.

Proposed change

1. Separate three allocation decisions

  • Storage capacity: how much physical GPU memory backs the engine-wide KV pool.
  • Request ownership: which committed KV blocks a request currently owns; allocate these incrementally as needed.
  • Logical addressing: metadata maps layer, loop and token identity to stored KV.

This RFC changes startup profiling and storage capacity. It does not require reserving all future output blocks for each request or organizing physical storage into preallocated depth planes. It should work with the independently discussed KV page-layout changes in #147.

GPU virtual-memory pages and token blocks are different units. Mapping alignment must be handled by storage; the request allocator only exposes fully backed, usable KV blocks.

2. Change startup order

load model and select backend
  -> reserve a stable virtual address range for KV
  -> commit enough physical storage for valid warmup inputs
  -> initialize the actual runner, buffers, metadata and enabled connectors
  -> warm the configured execution paths
  -> capture the planned graph entries and exercise replay
  -> measure remaining runtime transient requirements
  -> drain all startup work and release temporary requests/tensors
  -> determine and commit final KV capacity
  -> publish the usable block pool and accept requests

The virtual reservation has a bounded upper limit derived from supported capacity and device constraints. It does not consume that amount of physical VRAM.

Warmup uses scratch requests whose accessed KV addresses are mapped and valid. Masked/inactive rows must never access unmapped storage. Scratch state must not enter prefix caches, consume real-request RNG state, or survive into serving.

After final sizing, verify replay and a representative execution within the measured budget before declaring the engine ready.

3. Preserve captured addresses

Replace direct allocation of the final K/V tensors with storage that can add physical mappings inside a stable virtual range.

  • Captured pointers, tensor views and strides remain valid across physical commitment.
  • Runtime metadata controls valid block indices; reserved virtual capacity is never treated as committed capacity.
  • The logical allocator receives only committed blocks after sizing.
  • One storage owner manages mappings and tensor lifetimes. Unmap/free only after dependent device work and graph users have retired.
  • Account for these allocations explicitly: driver/VMM allocations may not appear in PyTorch allocator statistics.

This requires a GPU storage implementation, not simply reallocating or resizing a PyTorch tensor after capture. GPU VMM operations belong outside the request execution hot path.

4. Warm the paths enabled by the configuration

Warmup should trigger compilation, CUDA library initialization and actual workspace allocation, and establish a representative memory envelope. It is not a full benchmark suite.

Cover the relevant prefill token limits, decode batches and graph buckets, boundary-KV stages, packed/staged prefill, and speculative verification when enabled. Include context lengths that affect attention scratch usage. A single largest batch is insufficient when shapes select different kernels or workspaces.

For asynchronous/multi-stream execution, exercise supported overlap and buffer lifetimes. Measure the concurrent envelope rather than assuming it is twice a serial forward peak.

Use existing per-loop execution and scheduling boundaries. Warmup and capture must preserve mixed admission semantics.

5. Measure persistent usage and incremental transient demand

Record driver-free memory alongside PyTorch allocated/reserved/peak statistics at phase boundaries. Drain participating streams before each boundary measurement and apply a consistent temporary-object/cache cleanup policy.

Distinguish:

  • Persistent: weights, execution buffers, metadata, captured graphs and retained outputs.
  • Transient: additional physical memory needed during subsequent execution above that persistent baseline.
  • Bootstrap KV: mapped scratch storage, tracked separately from model workspace.

After cleanup, persistent allocations already reduce driver-free memory. Do not subtract graph memory again, or count temporary probe KV as runtime workspace. PyTorch caching and reusable workspace must be accounted for consistently so that the same bytes are not reserved twice.

Final KV capacity consists of retained bootstrap mappings plus additional mappings that fit within:

  1. measured remaining physical memory;
  2. measured additional runtime transient needs and a safety margin;
  3. the remaining configured instance-memory budget;
  4. supported/useful capacity and mapping alignment.

Keep gpu_memory_utilization as a cap; this RFC does not change its default to 1.0. Preserve explicit num_blocks / kv_cache_memory_bytes as capacity overrides with clear startup failure if the requested capacity cannot coexist with execution. Overrides may skip automatic sizing, but must not disable required capture preparation.

Log startup phase costs, graph entry coverage, persistent usage, transient allowance, safety margin, committed KV bytes and usable block count. Do not consume all post-capture free memory: later eager execution still needs working space.

6. Resolve lazy graph growth before removing graph headroom

For the measured path, define a bounded graph-entry plan from the configured limits and complete its capture before final sizing. Unplanned shapes use the existing eager fallback, whose workspace is included in warmup coverage.

Changing graph shape keys or bucket padding requires output/metadata validation, including retained output lifetimes and shared graph-pool replay ordering. A bucketed shape must not alter logical loop scheduling.

If a path still permits serving-time capture, retain an explicit bounded budget for it and report that coverage gap. Do not claim fully measured capacity or remove all graph headroom while future graph allocations remain unaccounted for.

Implementation boundaries and rollout

  1. Coordinate startup warmup and capture using the existing execution paths; add phase measurements. Retain the current allocation strategy while this is introduced.
  2. Add stable-address GPU KV storage, integrate final capacity publication, and validate it with capture/replay.
  3. Enable measured sizing on supported configurations. Retire fixed per-pool graph estimates where capture coverage is complete.

StageEngine coordinates initialization. Execution components own warmup/capture and device buffers; memory planning calculates capacity; storage owns mappings; the logical KV manager owns block leases and request mappings. Avoid a second scheduler or duplicate request-state model.

Select supported behavior automatically. Keep the existing bounded CPU behavior and an identified estimate-based fallback for unsupported VMM/platform/connector combinations. Validate connector registration/export requirements before selecting the storage path; report the fallback reason. Clean up partial mappings and graphs if startup fails.

Alternatives and expected impact

  • Keep fixed graph reserves: simple, but continues to rely on estimates that vary with enabled paths and retained outputs.
  • Allocate a small ordinary KV tensor, then replace it after capture: invalidates captured pointers; recapturing against a larger tensor reintroduces the capacity/capture dependency.
  • Use every byte free after capture: leaves no allowance for later transient execution.
  • Commit physical storage per request during serving: unnecessary for this startup proposal; requests can incrementally lease blocks from a committed pool.

Expected tradeoff: more predictable serving startup and potentially better use of VRAM, at the cost of more startup work and GPU storage complexity. The capacity gain is configuration-dependent and may be small or negative where existing estimates were insufficient.

Validation

  • Verify captured pointers/views/strides remain valid through commitment and that previously captured graphs replay using newly committed blocks.
  • Compare generated tokens/depths and KV contents against the existing path for boundary-KV and recurrent models, eager/graph, supported synchronous/asynchronous paths, mixed scheduling, and enabled speculation.
  • After sizing, run representative high-memory prefill, decode and supported overlaps; verify transient headroom is sufficient.
  • Exercise partial-startup failure, cleanup, repeated engine creation, block reuse and supported connector lifetimes.
  • Compare baseline/candidate startup duration, committed KV capacity, physical peak usage, first-request latency and steady-state performance under matched settings. Report limitations and untested combinations.

Preemption snapshot/recomputation remains tracked separately in #191.

References

Activity

  1. added
    RFCRequests for comments on major architectural changes or design choices
    on Nov 16, 2026
  2. kunkunblueberry commented on Nov 16, 2026

    @kunkunblueberry
    Contributor

    I'd like to contribute to the full runtime KV memory work across the three related PRs. These parts are tightly coupled, so I think it is important to understand and help implement the end-to-end path rather than treat capacity planning or storage initialization in isolation.

    In particular, I'm interested in:

    • Runtime memory measurement after warmup / CUDA Graph capture.
    • KV cache capacity planning based on the measured runtime memory budget.
    • KV storage reservation / initialization and its integration with KVCacheManager and ModelRunner.

    I'm happy to work across all three PRs and help keep the interfaces and lifecycle consistent. We can coordinate the concrete split of implementation work as the design develops.

  3. cecilsd commented on Nov 16, 2026

    @cecilsd
    Contributor

    I'd like to take the startup warmup and CUDA Graph capture orchestration part of step 1. I'll focus on running the existing warmup paths during StageEngine startup, capturing a bounded set of graph entries, and checking replay before the engine accepts requests. The first change will use the current KV allocation strategy.

    I'll coordinate with @kunkunblueberry on where warmup and capture hand off to memory measurement. KV sizing, stable-address storage, and final capacity publication are outside my scope.

  4. XinhaoTheo commented on Nov 16, 2026

    @XinhaoTheo

    I like to take on the VMM storage backend, reserving stable KV addresses and committing physical memory without invalidating captured graphs.

    @kunkunblueberry @cecilsd, would that split work for you? I will focus on the storage backend and its lifecycle, coordinating with your capacity planning and warmup/capture work on the interfaces.

  5. bjf-frz commented on Nov 17, 2026

    @bjf-frz
    CollaboratorAuthor

    Thanks all, assigned to @kunkunblueberry, @cecilsd and @XinhaoTheo. The split proposed here works:

    • @cecilsd: startup warmup and CUDA Graph capture orchestration (step 1), starting with the current KV allocation strategy.
    • @XinhaoTheo: VMM storage backend: stable KV address reservation, physical commitment without invalidating captured graphs, and its lifecycle.
    • @kunkunblueberry: runtime memory measurement after warmup/capture, KV capacity planning from the measured budget, and integration with KVCacheManager / ModelRunner, keeping the interfaces and lifecycle consistent across the three parts.

    Please agree on the hand-off interfaces (warmup/capture → measurement → storage commitment) here before the PRs depend on each other.

  6. XinhaoTheo commented on Nov 17, 2026

    @XinhaoTheo

    Thanks all, assigned to @kunkunblueberry, @cecilsd and @XinhaoTheo. The split proposed here works:

    • @cecilsd: startup warmup and CUDA Graph capture orchestration (step 1), starting with the current KV allocation strategy.
    • @XinhaoTheo: VMM storage backend: stable KV address reservation, physical commitment without invalidating captured graphs, and its lifecycle.
    • @kunkunblueberry: runtime memory measurement after warmup/capture, KV capacity planning from the measured budget, and integration with KVCacheManager / ModelRunner, keeping the interfaces and lifecycle consistent across the three parts.

    Please agree on the hand-off interfaces (warmup/capture → measurement → storage commitment) here before the PRs depend on each other.

    Agree

  7. cecilsd commented on Nov 18, 2026

    @cecilsd
    Contributor

    @kunkunblueberry @XinhaoTheo I've opened PR #198 for startup warmup and CUDA Graph capture orchestration, based on bed3400, following the agreed division of work. It retains the current KV allocation strategy and has no dependency on the other implementation branches. The following is a proposed handoff for us to agree on before integration.

    The worker now initializes/registers its connector, calls the idempotent StageEngine.prepare_runtime(), and sends ready only after preparation succeeds. Preparation uses the actual serving runners and physically valid scratch KV leases. Scratch requests do not enter the scheduler or publish prefix-cache entries. The successful return and cleaned startup_report provide the current step-1 handoff; they do not calculate or approve a new KV capacity.

    Measurement boundary and ownership

    • Participating streams and submission queues are drained before each phase measurement and before scratch ownership is released.
    • The report includes cumulative elapsed seconds, driver free/total bytes, PyTorch allocated/reserved/peak bytes, bounded workload coverage, and Graph keys/replay/fallback information. The initial peak includes earlier initialization; subsequent peaks start at the preceding phase reset. Driver-free values are boundary samples, not a continuous physical high-water mark.
    • Temporary requests, KV leases, request-local RNG, and async/speculative state are cleaned up; serving RNG and business counters are preserved. Graphs, execution buffers, and library workspaces remain alive. No empty_cache is called between phases (allocator_cache_policy="retain").
    • num_blocks and kv_bytes describe the currently allocated pool, including FP8 scales. They are not measured-sizing results. Measurement should account for KV separately and avoid subtracting already-resident Graph memory again.
    • Warmup/capture coverage is bounded and gaps are reported. Existing estimated sizing and reserves for future lazy capture remain in place.

    Proposed integration sequence

    physically backed scratch KV at stable addresses
      -> warmup / capture / representative replay
      -> drain and release temporary state
      -> memory measurement and capacity planning
      -> final physical storage commitment
      -> replay and representative execution validation
      -> publish usable blocks and readiness
    

    Storage needs to preserve captured pointers, tensor views, and strides across commitment, and publish only fully backed usable blocks. Failure cleanup must drain dependent device work before releasing mappings or owners. The current PR does not implement VMM, final measured sizing, or post-commit validation; those belong to the subsequent integration.

    Validation

    • Linux pre-commit run --all-files passed. The full CPU suite recorded 1236 passed / 35 skipped.
    • RTX 5090 startup/fixture coverage recorded 80 passed; other applicable regressions recorded 307 passed / 2 skipped. Dual-GPU skips and unsupported FA4 SM120 cases are disclosed in the PR.
    • Full Ouro-1.4B BF16 weights were checked across five execution modes and BF16/FP8 KV. Within each KV dtype, all 150 matched request cases preserved token IDs and exit depths; observed ordinary-sampler logits and valid KV content, including FP8 scales, matched exactly.
    • First-request TTFT fell from about 728 to 141 ms synchronously and 750 to 141 ms asynchronously in the documented workload. Startup increased to about 2.04 s / 3.18 s, and startup plus first completion became slower. No sustained throughput improvement is established. Independent ops-only/full profiles and measurement details are in the PR.

    Does this drained/cleaned handoff and report semantics fit your measurement/storage work? We should also agree on how StageEngine coordinates validation after final storage commitment, and on connector registration requirements for the mapping lifecycle, before fixing the cross-PR interfaces.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

RFCRequests for comments on major architectural changes or design choices

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions