Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
f6286f5
[KV] FP8 e4m3 KV cache with per-vector scales
Oct 15, 2026
ee375cb
[Docs] FP8 KV cache usage, semantics and limits
Oct 15, 2026
0e530b2
[KV] Fused FP8 quantize-and-store kernel for Triton
0z5a Oct 16, 2026
f36f2f9
[Docs] FP8 KV: fused Triton writer, current token is read in FP8
0z5a Oct 16, 2026
6087e67
[Fix] Merge main into FP8 KV cache support
0z5a Oct 21, 2026
b2803a2
Merge remote-tracking branch 'upstream/main' into kv/fp8-kv-cache
0z5a Nov 1, 2026
fbfeaad
Run the merged FP8 tests on GPUs with less shared memory
0z5a Nov 1, 2026
8baa95f
fix(attention): reject FP8 per-vector KV on FlashAttention with guidance
0z5a Nov 6, 2026
d2043f5
feat(attention): support FP8 per-vector KV on FA2-4
0z5a Nov 6, 2026
529151c
feat(attention): advertise FP8 KV on FA2-4
0z5a Nov 6, 2026
65163d4
fix(engine): reject CUDA graphs for FP8 KV on FlashAttention
0z5a Nov 6, 2026
9c81c31
fix(attention): keep FA3 paged FP8 disabled until validated
0z5a Nov 6, 2026
9fed631
perf(attention): fuse FP8 KV staging into one Triton launch
0z5a Nov 6, 2026
260404a
perf(attention): cache FP8 staging buffers and page-table map
0z5a Nov 6, 2026
c03f75e
Merge main and read FP8 KV scales in the windowed Torch reference
0z5a Nov 13, 2026
0f31f60
Cover FP8 scales in windowed Torch attention
0z5a Nov 13, 2026
7f001e1
Merge main and read FP8 KV scales in Triton and graph window paths
0z5a Nov 16, 2026
03bb389
Cover FP8 scales in Triton and graph-replayed window attention
0z5a Nov 16, 2026
9ed813e
Merge main and preserve FP8 scales in packed window attention
0z5a Nov 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions docs/runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -325,6 +325,31 @@ zero watermark in non-incremental mode, see [scheduler admission](scheduler.md#a
Pool sizing establishes physical capacity; scheduler admission decides how much
of that capacity can safely be promised to new work.

## FP8 KV cache

`CacheConfig(kv_cache_dtype="fp8_e4m3")` (CLI `--kv-cache-dtype fp8_e4m3`) stores K and V as
float8 e4m3 codes with one float32 scale per (block, layer, token, KV head) vector
(`scale = amax / 448`). Values are quantized when written and dequantized by the attention
backend when read, so a block costs `head_dim + 4` bytes per vector instead of
`head_dim * element_size`: 51.6% of BF16 at `head_dim=128`, and automatic sizing or
`--kv-cache-memory-bytes` admits about 1.94x as many tokens. The default `"auto"` stores the
model dtype and leaves every existing path unchanged.

- Supported: Torch (reference) and Triton attention; synchronous and asynchronous scheduling;
CUDA graphs (quantized writes are captured); LAST_EXITED finalize copies, including the
asynchronous finalize kernel, copy codes and scales; preemption snapshots carry the scales;
`KVCacheManager.read` returns dequantized values.
- Triton's split-KV decode and packed paged-prefill kernels both apply the scales on load
(prefill dequantizes to the query dtype before `tl.dot`). Device-masked writes (speculative
verification, bucketed CUDA graphs) use the same fused writer with the row mask applied.
- Triton writes with one fused quantize-and-store kernel per layer. The Torch backend quantizes
with Torch ops and is the reference; both writers produce the same codes and scales.
- Rejected: FlashAttention backends (the `fp8_kv` backend capability; `auto` selects Triton)
and PD transfer, which does not yet move the scales.
- Each layer writes the current token's K/V before attending, so attention reads the current
token in FP8 as well. Accuracy should be checked per model and workload; adaptive exits can
change where a token exits because the gate sees slightly different attention.

## Replay and validation commands

Trace mode expects `depths_by_request`, with one depth per output: full prefill depth first, then each decode input's exit depth. Admission validates coverage and loop bounds. Request IDs must match the trace. The public engine replay fixes depth decisions; token generation still follows the configured sampler.
Expand Down
3 changes: 2 additions & 1 deletion tests/test_cuda_graph.py
Original file line number Diff line number Diff line change
Expand Up @@ -455,7 +455,8 @@ def test_bucketed_kv_graph_combines_active_rows_and_pipeline_write_mask():

keys = torch.full((4, 1, 2, 1, 4), -1.0, device="cuda")
values = torch.full_like(keys, -2.0)
cache = SimpleNamespace(key_cache=keys, value_cache=values)
# A BF16 pool has no FP8 scales, as KVCacheManager reports them.
cache = SimpleNamespace(key_cache=keys, value_cache=values, key_scale=None, value_scale=None)
batch = SimpleNamespace(
active_rows=torch.zeros((), dtype=torch.int32, device="cuda"),
write_mask=torch.ones(5, dtype=torch.bool, device="cuda"),
Expand Down
10 changes: 8 additions & 2 deletions tests/test_dory_gpu.py
Original file line number Diff line number Diff line change
Expand Up @@ -449,9 +449,10 @@ def compile_tile(*args, window, **kwargs):
assert probe(index, 256, 16, PackedWindowShape(16, 4, 36, 1024)) is None


@pytest.mark.parametrize("fp8_kv", [False, True])
@pytest.mark.parametrize(("window", "expected_windows"), [(None, [0]), (128, [0, 128])])
def test_prefill_tile_probe_compiles_only_the_default_and_caches_result(
monkeypatch, window, expected_windows
monkeypatch, window, expected_windows, fp8_kv
):
from types import SimpleNamespace

Expand All @@ -470,12 +471,17 @@ def test_prefill_tile_probe_compiles_only_the_default_and_caches_result(
def compile_tile(*args, **kwargs):
assert (kwargs["BLOCK_M"], kwargs["BLOCK_N"]) == (32, 32)
assert kwargs["COMPENSATE"]
assert kwargs["HAS_SCALE"] is fp8_kv
assert args[1].dtype == (torch.float8_e4m3fn if fp8_kv else torch.bfloat16)
if fp8_kv:
assert args[7].dtype == args[8].dtype == torch.float32
assert args[7].shape == args[1].shape[:-1]
assert all(arg.device.type == "meta" for arg in args if isinstance(arg, torch.Tensor))
windows.append(kwargs["WINDOW"])
return SimpleNamespace(metadata=SimpleNamespace(shared=limit))

monkeypatch.setattr(attention._paged_prefill_kernel, "warmup", compile_tile)
args = (device, torch.bfloat16, 64, 16, PackedWindowShape(4, 2, 2, window))
args = (device, torch.bfloat16, 64, 16, PackedWindowShape(4, 2, 2, window, fp8_kv))

attention._select_packed_window_tile.cache_clear()
try:
Expand Down
Loading
Loading