Skip to content

[RFC] Make preemption snapshots asynchronous and compare with recomputation #191

Open

Description

@bjf-frz

Problem

At main c338aaa, capacity-pressure preemption blocks the engine's scheduling path:

Enabling async scheduling does not make this recovery path asynchronous. This is established by code inspection; no snapshot-versus-recompute benchmark has been run for this issue.

Required change: asynchronous snapshot/offload and restore

Move transfers off the blocking scheduling path:

  • Use bounded, reusable pinned host buffers and an asynchronous transfer stream. Wait on the victim's producer events instead of draining the whole runner. Batch transfers where practical without allocating a request-sized GPU temporary under memory pressure.
  • Keep source blocks and borrowed state leased until D2H completes. Only then may the allocator reuse those blocks. Other eligible requests should continue while the copy is pending.
  • Restore into reserved destination blocks; mark the request executable only after H2D completion and the required stream dependencies.
  • Preserve KV validity at every relevant layer/depth, prelude/coda history, hidden/token state, execution progress and request-local RNG. Preserve supported speculative state without broadening currently unsupported combinations.
  • Define cancellation, failure and request-ID reuse handling so late completions cannot release or overwrite another request's resources. Bound pinned-memory use and pending transfers.

Merely adding non_blocking=True is insufficient: ownership, completion and scheduler readiness must follow the transfer lifecycle. Async offload also cannot make a victim's blocks available before its copy completes.

Discuss snapshot/offload versus recomputation

Compare the critical-path cost of D2H + H2D + synchronization/bookkeeping against recomputing the KV that cannot be reused. Include prefix-cache hits, sequence length, loop depth, KV bytes, effective transfer bandwidth, batching and overlap with unaffected requests.

For looped models, recomputation must reproduce the historical execution depths and boundary-KV semantics; replaying all committed tokens as an ordinary full-depth prompt is not automatically equivalent. Choose recovery defaults from matched measurements rather than assuming either strategy always wins.

References: vLLM V1 preemption, the prefix-cache and legacy-swap rationale, and asynchronous KV offloading. KV-cache offloading is distinct from a complete execution-state snapshot.

Validation and scope

Exercise forced KV pressure, resume, cancellation, failure and ID reuse, with async execution and CUDA Graphs where supported. Verify state/output correctness and eventual resource cleanup. Use matched unprofiled latency/throughput measurements plus small ops-only/full profiles to show transfer overlap, block-release timing and impact on unaffected requests; report host-memory use and bytes transferred.

This is a separate follow-up identified during #147 review. Keep recovery-policy discussion and implementation out of #147, which remains focused on KV layout/allocation.

Activity

  1. GoDiao commented on Nov 16, 2026

    @GoDiao

    @bjf-frz I'd like to take this — could you assign it to me?

    Planned direction, in 2 steps:

    1. Async offload/restore. Move the KV copies off the scheduling path onto a transfer stream with bounded pinned buffers, wait only on the victim's own work instead of draining the runner, and tie block release and scheduler readiness to transfer completion. I'll look at reusing the existing transfer-lease path in KVCacheManager for ownership.
    2. Snapshot vs. recompute. Matched measurements on top of step 1 to choose the recovery default.

    Would you prefer these as two separate PRs, or one?

    I'll keep this clear of the layout work in #147 and coordinate with #164 / #98, which extend the same snapshot record.

  2. bjf-frz commented on Nov 17, 2026

    @bjf-frz
    CollaboratorAuthor

    @GoDiao Thanks, assigned to you. The plan looks good.

    Please split it into two PRs: the async offload/restore first, then the snapshot-vs-recompute measurements on top of it, so the measurements run against the merged step 1. Keeping clear of #147 and coordinating with #164 / #98 on the snapshot record sounds right.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions