You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
perf(server): keep CUDA graph replay when LoRA adapters are loaded #840
Loading a LoRA adapter takes a model off CUDA graph replay, and on short encoder calls the eager path is several times slower.
On main, the ModernBERT and ColBERT flash adapters run every request eagerly once any PEFT LoRA is loaded, base-model requests included (packages/sie_server/src/sie_server/adapters/modernbert_flash/__init__.py: if self._graphs is None or self._peft_model is not None: return None). For gte-modernbert-base on an L4, graphs are worth 17.5 → 2.4 ms per query (perf(server): replay ModernBERT flash forwards as CUDA graphs #423).
Requests that select a LoRA adapter replay recorded graphs as base requests do, under the same numerics rule as the other graph work: no probability moves more than 0.02 against eager, or the result is no worse against float32.
Possible approaches
Static adapter buffers. Capture graphs with the LoRA A/B matrices read from fixed buffers. Before replay, copy the selected adapter's weights into those buffers, so adapters swap without a recapture. Needs one graph set per rank and target-module layout.
Per-adapter recordings for hot adapters. Record a graph set per (adapter, bucket) inside the existing graph memory budget, and evict a recording together with its adapter in the LoRA LRU.
Merge at staging. Merge a hot adapter into a copy of the base weights and record graphs for that copy. Simplest numerically, but it costs memory per adapter.
Whichever approach is chosen, base-model graphs must stay unaffected when adapters are loaded.
Acceptance
Base requests keep graph replay while any adapter is loaded, starting with the ModernBERT and ColBERT flash adapters.
Adapted requests replay graphs on GLiNER2.5-Decide and at least one ModernBERT model. Report parity numbers and L4 p50 before and after.
Graph memory stays within the existing budget, and evicting an adapter frees its recordings.
Activity