feat(server): keep evicted models in host RAM and wake them instead of reloading

deploy/helm/sie-cluster/README.md

@@ -78,6 +78,18 @@ The current GKE/EKS scale-from-zero guidance is:
 | **Model loading** | 10-120s | Download weights (if not cached), load to GPU |
 | **Total cold start** | 3-7 min | First request to a scaled-to-zero pool |
 
+Model loading also applies when a running worker reloads a model it evicted
+to make room for another. Workers keep evicted models in host RAM and wake them
+back onto the GPU instead, which is estimated to take well under a second for
+a typical encoder and a few seconds for a single-GPU SGLang model. The figures
+are estimates, not measurements. A model loads the ordinary way when it was
+never loaded on that worker, or when it was dropped from host RAM to stay
+within the budget. Under the default `SIE_HOST_CACHE_BYTES=auto` each GPU
+worker container uses a quarter of its memory limit, 8 GiB on the default L4
+pool. Override it with `workers.common.extraEnv` or a bundle's `extraEnv`, or
+set it to `0` to turn the host tier off. See the server README section "Host-RAM tier for evicted
+models" for what parks and how it is charged.
+
 These figures are not an ACK performance claim; this repository has no ACK
 cold-start baseline. The Alibaba Terraform module defaults its shared NAT EIP
 to 100 Mbit/s, and observed ACK cold start depends on current GPU inventory and
@@ -86,9 +98,10 @@ quota, effective bandwidth, image and model size, and cache state.
 ### Reducing Cold Start Time
 
 1. **Use cluster cache**: Pre-populate object storage with model weights (`--cluster-cache`)
-2. **Set minReplicas=1**: Keep one warm replica per critical GPU type
-3. **Use reserved capacity**: Avoid spot for latency-sensitive workloads
-4. **Pre-warm models**: Call `/v1/encode/{model}` on startup to load weights
+2. **Size the host-RAM tier**: Raise `SIE_HOST_CACHE_BYTES` (with the pool's memory limit) so models that take turns on a GPU wake from host RAM instead of reloading
+3. **Set minReplicas=1**: Keep one warm replica per critical GPU type
+4. **Use reserved capacity**: Avoid spot for latency-sensitive workloads
+5. **Pre-warm models**: Call `/v1/encode/{model}` on startup to load weights
 
 ### Client Handling
 

deploy/helm/sie-cluster/files/dashboards/worker-health.json

@@ -468,6 +468,56 @@
           "refId": "B"
         }
       ]
+    },
+    {
+      "gridPos": { "h": 10, "w": 12, "x": 0, "y": 78 },
+      "id": 76,
+      "title": "Host-RAM Tier Lookups and Wake Time",
+      "description": "Model loads that woke a copy parked in host RAM (hit) or loaded from disk, the cluster cache or the Hub (miss), and the p95 time to wake a parked model back onto its device.",
+      "type": "timeseries",
+      "fieldConfig": { "defaults": { "unit": "short" } },
+      "options": {
+        "legend": { "calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true },
+        "tooltip": { "mode": "multi", "sort": "desc" }
+      },
+      "pluginVersion": "10.0.0",
+      "targets": [
+        {
+          "expr": "sum by (result, lane) (rate(sie_worker_model_host_cache_lookups_total{namespace=\"$namespace\", service=\"$collector\", endpoint=\"prometheus\", producer_service=\"sie-worker\"}[5m]))",
+          "legendFormat": "loads/sec {{result}} / {{lane}}",
+          "refId": "A"
+        },
+        {
+          "expr": "histogram_quantile(0.95, sum by (le, lane) (rate(sie_worker_model_host_cache_wake_duration_seconds_bucket{namespace=\"$namespace\", service=\"$collector\", endpoint=\"prometheus\", producer_service=\"sie-worker\", outcome=\"success\"}[5m])))",
+          "legendFormat": "wake p95 seconds / {{lane}}",
+          "refId": "B"
+        },
+        {
+          "expr": "sum by (outcome, lane) (rate(sie_worker_model_host_cache_wake_duration_seconds_count{namespace=\"$namespace\", service=\"$collector\", endpoint=\"prometheus\", producer_service=\"sie-worker\"}[5m]))",
+          "legendFormat": "wakes/sec {{outcome}} / {{lane}}",
+          "refId": "C"
+        }
+      ]
+    },
+    {
+      "gridPos": { "h": 10, "w": 12, "x": 12, "y": 78 },
+      "id": 77,
+      "title": "Host-RAM Tier Memory",
+      "description": "Host RAM the host tier holds for parked models (and for SGLang engines that keep their host copy after waking), by model.",
+      "type": "timeseries",
+      "fieldConfig": { "defaults": { "unit": "bytes" } },
+      "options": {
+        "legend": { "calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true },
+        "tooltip": { "mode": "multi", "sort": "desc" }
+      },
+      "pluginVersion": "10.0.0",
+      "targets": [
+        {
+          "expr": "max by (model, profile, lane) (sie_worker_model_host_cache_memory_bytes{namespace=\"$namespace\", service=\"$collector\", endpoint=\"prometheus\", producer_service=\"sie-worker\"}) > 0",
+          "legendFormat": "{{model}} / {{profile}} / {{lane}}",
+          "refId": "A"
+        }
+      ]
     }
   ],
   "refresh": "10s",

packages/sie_server/README.md

@@ -4,7 +4,8 @@ GPU inference server for embeddings, reranking, and entity extraction.
 
 ## Features
 
-- Multi-model serving with LRU eviction
+- Multi-model serving with LRU eviction, and a host-RAM tier that wakes evicted
+  models back onto the GPU instead of reloading them
 - Token-based dynamic batching
 - Hot reload model configs without restart
 - Unified API: `encode()`, `score()`, `extract()`
@@ -1228,6 +1229,71 @@ When OOM recovery is exhausted on a request, the server returns
 `HTTP 503 RESOURCE_EXHAUSTED` with `Retry-After`. The Python SDK
 auto-retries; see `packages/sie_sdk/README.md` for client-side controls.
 
+### Host-RAM tier for evicted models
+
+When the worker evicts a model to free GPU memory, it keeps the model's weights
+in host RAM instead of discarding them, up to a budget. The next request for
+the model copies the weights back to the same GPU. It does not rebuild the
+adapter, re-read the local disk cache, or fetch from the cluster cache or
+Hugging Face. When the budget fills, the least recently parked model is
+unloaded first. A model that is not parked loads the ordinary way.
+
+| Env var | Default | Effect |
+|--|--|--|
+| `SIE_HOST_CACHE_BYTES` | `auto` | Host RAM the tier may hold. `auto` uses a quarter of the container memory limit (cgroup v2 or v1), or of physical RAM when there is no limit, on CUDA workers, and is off on CPU and Apple silicon. A size such as `8GiB`, `8Gi` or `8000000000` sets it explicitly. `0` turns the tier off. |
+
+With `auto`, an L4 pool with the chart's 32Gi container limit gets 8 GiB, and
+the 128Gi and 256Gi A100 and H200 pools get 32 GiB and 64 GiB.
+
+What parks:
+
+- Evictions that free memory for other work park the model: memory pressure,
+  making room for another load, OOM recovery, idle eviction
+  (`SIE_IDLE_EVICT_S`) and displacement by a multi-GPU model. A manual unload,
+  a config change, shutdown and a crashed engine unload the model as before.
+  Pinned models are never evicted, so they never park.
+- Adapters that hold their PyTorch model in the worker process
+  (sentence-transformers, the flash encoders, cross-encoders and the other
+  in-process adapters) park on CUDA by copying every tensor the adapter
+  references to ordinary host memory and freeing the GPU copies. Parameters keep their identity, so LoRA adapters loaded through PEFT
+  park and wake with the base model. CUDA graphs are dropped before parking and
+  record again on demand. An adapter declines when it holds GPU tensors the
+  copy cannot rebind or quantized parameters.
+- SGLang generation and embedding engines on one GPU park through SGLang's
+  memory saver. SIE launches these engines with `--enable-memory-saver
+  --enable-weights-cpu-backup` while the tier is on. Parking waits up to 10
+  seconds for in-flight requests and aborts any that remain, then releases the
+  weights, KV cache and CUDA graph memory. The weights go to pinned host memory.
+  The engine process, its compiled kernels and its captured graphs stay alive,
+  so a wake maps memory back at the same addresses, copies the weights, and
+  checks that the engine generates. The memory saver in the pinned SGLang keeps
+  that host copy after a wake, so an engine that has parked once keeps its weights'
+  size charged to the budget until it is fully unloaded. A parked engine also
+  keeps its CUDA context and some workspace on the GPU. The worker drops it
+  when that GPU needs the memory and no resident model can be evicted. Set
+  `adapter_options.loadtime.host_offload: false` on a profile to launch its
+  engine without the memory saver.
+- Tensor-parallel engines, TensorRT-LLM, MLX, CTranslate2, the vision
+  extraction engine, Candle workers and remote profiles unload as before.
+
+A request for a parked model waits up to 15 seconds for the wake instead of
+receiving `503 MODEL_LOADING`. A queued request is held by the sidecar, which
+checks readiness every second. A wake that fails falls back to an ordinary
+load. A wake runs one at a time with loads, so it can wait behind another
+model's cold load.
+
+Wake time is bounded by the host-to-GPU copy. These are estimates, not
+measurements: a 1 GB encoder should wake in about 0.1 to 0.3 seconds from
+ordinary host memory. An 8 GB LLM should take about 1 to 2 seconds from pinned
+memory, plus the engine's one-token health check. A load from the local disk
+cache takes seconds to minutes, because it constructs the model, warms it up
+and, for SGLang, starts an engine and captures its CUDA graphs.
+
+The worker reports `sie.worker.model.host_cache.lookups` (each load, `hit` or
+`miss`), `sie.worker.model.host_cache.wake.duration` and
+`sie.worker.model.host_cache.memory` (host bytes per model); see
+`telemetry/contract.yaml`.
+
 ### Batching & request handling
 
 | Env var | Default | Effect |

packages/sie_server/src/sie_server/adapters/_base_adapter.py

@@ -10,7 +10,7 @@
 from sie_server.adapters._spec import AdapterSpec
 from sie_server.adapters._types import ERR_NOT_LOADED
 from sie_server.adapters._utils import grouped_score_pairs
-from sie_server.adapters.base import ModelAdapter, ModelCapabilities, ModelDims
+from sie_server.adapters.base import HostOffload, ModelAdapter, ModelCapabilities, ModelDims, released_bytes
 
 if TYPE_CHECKING:
     import torch
@@ -36,6 +36,12 @@ class BaseAdapter(ModelAdapter):
 
     spec: ClassVar[AdapterSpec]
 
+    # Host-RAM parking (``ModelAdapter.offload_to_host``) for in-process
+    # weights. The generic implementation below moves every tensor reachable
+    # from the adapter's attributes (see ``_host_offload``). An adapter that
+    # keeps device state that walk cannot see sets this to False.
+    host_offload: ClassVar[bool] = True
+
     # Guards the lazy creation of each instance's ``_tokenizer_lock`` (below).
     # A HuggingFace *fast* tokenizer wraps a Rust object behind a ``RefCell``:
     # calling it from two threads at once — or mutating shared state like
@@ -127,6 +133,7 @@ def unload(self) -> None:
                 setattr(self, attr, None)
 
         self._device = None
+        self._parked_tensors = None
 
         gc.collect()
 
@@ -138,6 +145,99 @@ def unload(self) -> None:
             elif str(device) == "mps":
                 _torch.mps.empty_cache()
 
+    # -- Host-RAM parking ------------------------------------------------------
+
+    def _host_offload_device(self) -> torch.device | None:
+        """The CUDA device this adapter's weights live on, or None if parking would free nothing.
+
+        On CPU the weights already are in host RAM, and on Apple silicon the
+        accelerator shares it.
+        """
+        device = getattr(self, "_device", None)
+        if device is None:
+            return None
+        import torch as _torch
+
+        try:
+            resolved = _torch.device(device)
+        except (RuntimeError, TypeError, ValueError):
+            return None
+        return resolved if resolved.type == "cuda" else None
+
+    def host_offload_bytes(self) -> int | None:
+        """Bytes of device tensors the generic walk would park, or None when it cannot park."""
+        if not self.host_offload or getattr(self, "_parked_tensors", None) is not None:
+            return None
+        device = self._host_offload_device()
+        if device is None:
+            return None
+        if any(not callable(getattr(holder, "clear", None)) for holder in self._graph_holders()):
+            # A graph holder that cannot be cleared would replay against
+            # addresses the park frees.
+            return None
+        from sie_server.adapters import _host_offload
+
+        parkable = _host_offload.parkable_bytes(self, device)
+        # The model ``memory_footprint`` measures must be among what the walk
+        # reached; a shortfall means weights sit where parking cannot see them
+        # and would stay on the device. The slack covers parameters that share
+        # storage, which the footprint counts once each.
+        if parkable is None or parkable < 0.9 * released_bytes(self.memory_footprint()):
+            return None
+        return parkable
+
+    def offload_to_host(self) -> HostOffload:
+        """Copy the weights to host RAM and free their device memory."""
+        device = self._host_offload_device()
+        if device is None or getattr(self, "_parked_tensors", None) is not None:
+            msg = f"{type(self).__name__} has nothing on a CUDA device to park"
+            raise RuntimeError(msg)
+        from sie_server.adapters import _host_offload
+
+        self._drop_device_caches()
+        parked = _host_offload.park(self, device)
+        self._parked_tensors = parked
+        gc.collect()
+        _host_offload.release_cached_device_memory(device)
+        return HostOffload(host_bytes=parked.nbytes)
+
+    def restore_from_host(self) -> int:
+        """Copy parked weights back to the device they came from."""
+        parked = getattr(self, "_parked_tensors", None)
+        if parked is None:
+            msg = f"{type(self).__name__} is not parked"
+            raise RuntimeError(msg)
+        from sie_server.adapters import _host_offload
+
+        _host_offload.unpark(parked)
+        self._parked_tensors = None
+        return 0
+
+    def _drop_device_caches(self) -> None:
+        """Drop device state recorded against the current weight addresses.
+
+        A CUDA graph replays kernels against the exact addresses it recorded,
+        and parking moves every weight. Graph runners (``self._graphs``) are
+        cleared, and anything else the adapter releases as optional memory
+        goes with them; graphs record again on demand after the model wakes.
+        """
+        for holder in self._graph_holders():
+            holder.clear()
+        if self.has_releasable_memory():
+            self.release_optional_memory()
+
+    def _graph_holders(self) -> list[Any]:
+        """Attributes holding recorded CUDA graphs: runners (``_graphs``) and dicts of them.
+
+        Found by name, as every graph runner in this package is held. Scalars
+        that merely describe graphs (``_cuda_graphs = "off"``) are skipped.
+        """
+        return [
+            value
+            for name, value in vars(self).items()
+            if "graph" in name.lower() and value is not None and not isinstance(value, (bool, int, float, str))
+        ]
+
     def get_preprocessor(self) -> Any:
         """Return ``CharCountPreprocessor`` for cost estimation."""
         from sie_server.core.preprocessor import CharCountPreprocessor