Skip to content

perf(server): keep CUDA graph replay when LoRA adapters are loaded #840

Open

Description

@svonava

Problem

Loading a LoRA adapter takes a model off CUDA graph replay, and on short encoder calls the eager path is several times slower.

Goal

Requests that select a LoRA adapter replay recorded graphs as base requests do, under the same numerics rule as the other graph work: no probability moves more than 0.02 against eager, or the result is no worse against float32.

Possible approaches

  1. Static adapter buffers. Capture graphs with the LoRA A/B matrices read from fixed buffers. Before replay, copy the selected adapter's weights into those buffers, so adapters swap without a recapture. Needs one graph set per rank and target-module layout.
  2. Per-adapter recordings for hot adapters. Record a graph set per (adapter, bucket) inside the existing graph memory budget, and evict a recording together with its adapter in the LoRA LRU.
  3. Merge at staging. Merge a hot adapter into a copy of the base weights and record graphs for that copy. Simplest numerically, but it costs memory per adapter.

Whichever approach is chosen, base-model graphs must stay unaffected when adapters are loaded.

Acceptance

  • Base requests keep graph replay while any adapter is loaded, starting with the ModernBERT and ColBERT flash adapters.
  • Adapted requests replay graphs on GLiNER2.5-Decide and at least one ModernBERT model. Report parity numbers and L4 p50 before and after.
  • Graph memory stays within the existing budget, and evicting an adapter frees its recordings.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions