Repository navigation
Description
Problem
At main c338aaa, capacity-pressure preemption blocks the engine's scheduling path:
- PreemptionManager.suspend() synchronizes the runner before saving state.
- KVCacheManager.snapshot() / restore() use ordinary CPU buffers and per-block copies without nonblocking transfers.
- Resume synchronizes the current CUDA stream before returning.
Enabling async scheduling does not make this recovery path asynchronous. This is established by code inspection; no snapshot-versus-recompute benchmark has been run for this issue.
Required change: asynchronous snapshot/offload and restore
Move transfers off the blocking scheduling path:
- Use bounded, reusable pinned host buffers and an asynchronous transfer stream. Wait on the victim's producer events instead of draining the whole runner. Batch transfers where practical without allocating a request-sized GPU temporary under memory pressure.
- Keep source blocks and borrowed state leased until D2H completes. Only then may the allocator reuse those blocks. Other eligible requests should continue while the copy is pending.
- Restore into reserved destination blocks; mark the request executable only after H2D completion and the required stream dependencies.
- Preserve KV validity at every relevant layer/depth, prelude/coda history, hidden/token state, execution progress and request-local RNG. Preserve supported speculative state without broadening currently unsupported combinations.
- Define cancellation, failure and request-ID reuse handling so late completions cannot release or overwrite another request's resources. Bound pinned-memory use and pending transfers.
Merely adding non_blocking=True is insufficient: ownership, completion and scheduler readiness must follow the transfer lifecycle. Async offload also cannot make a victim's blocks available before its copy completes.
Discuss snapshot/offload versus recomputation
Compare the critical-path cost of D2H + H2D + synchronization/bookkeeping against recomputing the KV that cannot be reused. Include prefix-cache hits, sequence length, loop depth, KV bytes, effective transfer bandwidth, batching and overlap with unaffected requests.
For looped models, recomputation must reproduce the historical execution depths and boundary-KV semantics; replaying all committed tokens as an ordinary full-depth prompt is not automatically equivalent. Choose recovery defaults from matched measurements rather than assuming either strategy always wins.
References: vLLM V1 preemption, the prefix-cache and legacy-swap rationale, and asynchronous KV offloading. KV-cache offloading is distinct from a complete execution-state snapshot.
Validation and scope
Exercise forced KV pressure, resume, cancellation, failure and ID reuse, with async execution and CUDA Graphs where supported. Verify state/output correctness and eventual resource cleanup. Use matched unprofiled latency/throughput measurements plus small ops-only/full profiles to show transfer overlap, block-release timing and impact on unaffected requests; report host-memory use and bytes transferred.
This is a separate follow-up identified during #147 review. Keep recovery-policy discussion and implementation out of #147, which remains focused on KV layout/allocation.
Activity
GoDiao commentedon Nov 16, 2026
@bjf-frz I'd like to take this — could you assign it to me?
Planned direction, in 2 steps:
KVCacheManagerfor ownership.Would you prefer these as two separate PRs, or one?
I'll keep this clear of the layout work in #147 and coordinate with #164 / #98, which extend the same snapshot record.
bjf-frz commentedon Nov 17, 2026
@GoDiao Thanks, assigned to you. The plan looks good.
Please split it into two PRs: the async offload/restore first, then the snapshot-vs-recompute measurements on top of it, so the measurements run against the merged step 1. Keeping clear of #147 and coordinating with #164 / #98 on the snapshot record sounds right.