Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 41 additions & 1 deletion docs/dory.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,11 +90,51 @@ layer id rather than their physical sublayer index.
template, resolved without requiring transformers to accept the checkpoint
config (its generic validator rejects Dory's `mlp` sublayer entries).

## Packed Triton prefill

With `attention_backend="triton"`, the default `packed_prefill=None` automatically
enables packed prefill when execution and backend support it. Set
`ExecutionConfig(packed_prefill=True)` / `--packed-prefill` to require support,
or `packed_prefill=False` / `--no-packed-prefill` to keep per-query prefill.
An unsupported explicit enable request fails at startup.

This path uses BF16, `per_loop` KV, ordinary prefill, and synchronous eager
single-stage execution. Speculation, preemption, CUDA graphs and asynchronous
scheduling are outside its scope.

On NVIDIA CUDA, the packed path uses a 32 × 32 tile. At startup, Triton compiles
the full-attention kernel and, if the model uses a sliding window, the
sliding-window kernel. It checks their shared-memory requirements against the
device limit. Automatic mode uses per-query attention if the check fails;
explicit enable requests fail through backend capability validation.
The result is cached by device and attention geometry within the process.
The check runs before KV memory planning and uses tensor metadata only.
The check establishes that the tile fits, not that it is fastest. The fixed
32 × 32 tile was validated on RTX 4060 and RTX 5090; other GPUs and workloads
need measurements to determine their best tile.
The full BF16 checkpoint and long-context service validation require enough
device memory for the model weights, runtime buffers and KV pool.

A batch uses packed attention across PRELUDE, recurrence and the CODA
writes for prompt history when every prompt is longer than 64 tokens.
The CODA that samples the first token uses per-query attention. A batch containing
a shorter prompt or a 64-token prompt uses per-query attention. The
cutoff uses the full prompt length. Decode keeps the per-query path.

Packed full and sliding-window attention use the same causal ranges as
per-query attention. The matrix products and softmax reductions use a
different order and intermediate rounding. The value product compensates
for the BF16 rounding of softmax weights. Remaining differences can
propagate through Dory's recurrent layers and change a greedy choice.
For reproducible comparisons, use per-query attention and hold the
batch shape fixed. Changes in arrival timing can change the batch shape
even with per-query attention.

## Rejected today

These fail fast rather than degrade silently:

- WCPB, mixed or staged prefill; packed prefill metadata (`packed_prefill`); PD.
- WCPB, mixed or staged prefill, and PD.
- Prefix caching.
- Self-speculation combined with CUDA Graphs, asynchronous scheduling,
preemption or `last_loop` KV, and sampled token-pipeline speculation. The
Expand Down
11 changes: 8 additions & 3 deletions docs/runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,12 +81,17 @@ For a task-assigned GPU, set `CUDA_VISIBLE_DEVICES` and add `--device cuda --att
`ExecutionConfig.packed_prefill` controls packed prefill attention and metadata
staging. Its default, `None`, selects packed prefill when the selected backend
supports it on CUDA with `last_exited` KV and ordinary, non-staged prefill.
Mixed scheduling, wavefront chunked prefill and sliding-window attention keep
their existing paths. These selection rules apply to all models.
Mixed scheduling and wavefront chunked prefill keep their existing paths.
Sliding-window packing additionally requires backend support for both packed
prefill and packed sliding-window attention, with ordinary synchronous eager
single-stage execution, without speculation or preemption. These selection rules
apply to all models. Incompatible automatic configurations keep per-query attention.

Set `packed_prefill=True` / `--packed-prefill` to require packed prefill, or
`packed_prefill=False` / `--no-packed-prefill` to disable its staging path.
Explicit requests still undergo device, model and backend compatibility checks.
Explicit enable requests fail if execution constraints or required backend
capabilities are unsupported; they do not silently fall back. Explicit disable
does not probe a packed sliding-window tile.

`ExecutionConfig.use_uva` independently controls how input buffers are read.
`use_uva=False` / `--no-uva` copies them to the GPU and can be combined with
Expand Down
225 changes: 224 additions & 1 deletion tests/test_backend_selection.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ def import_module(name):
if name == "vllm_rlt.kernels.triton_attention":
if "triton" not in installed:
raise ImportError("triton is not installed")
return SimpleNamespace()
return SimpleNamespace(select_packed_window_tile=lambda *args: None)
if name in ("flash_attn", "flash_attn_3.flash_attn_interface", "flash_attn.cute"):
if name not in installed:
raise ImportError(f"{name} is not installed")
Expand Down Expand Up @@ -309,6 +309,62 @@ def test_explicit_triton_rejects_unsupported_nvidia_hardware(packages, monkeypat
create_backend("triton", torch.device("cuda"), torch.float16, 128, 256)


def test_triton_advertises_packed_window_only_after_selecting_a_tile(packages, monkeypatch):
from vllm_rlt.attention import backend as selection
from vllm_rlt.kernels.paged_attention import PackedWindowShape

monkeypatch.setattr(torch.cuda, "get_device_capability", lambda device: (8, 9))
base_import = selection.importlib.import_module
shape = PackedWindowShape(16, 4, 36, 1024)
calls = []

def probe(device, dtype, head_dim, block_size, requested_shape):
calls.append((device, dtype, head_dim, block_size, requested_shape))
return (32, 32)

def import_module(name):
if name == "vllm_rlt.kernels.triton_attention":
return SimpleNamespace(select_packed_window_tile=probe)
return base_import(name)

monkeypatch.setattr(selection.importlib, "import_module", import_module)
args = (torch.device("cuda"), torch.bfloat16, 256, 16)
backend = create_backend(
"auto",
*args,
required_capabilities=BackendCapabilities(sliding_window=True),
packed_window_shape=shape,
)
assert backend.info["selected_backend"] == "triton"
assert backend.info["packed_prefill_tile"] == (32, 32)
assert backend.capabilities.packed_prefill and backend.capabilities.packed_sliding_window
assert not create_backend("triton", *args).capabilities.packed_sliding_window
assert calls == [(*args, shape)]


@pytest.mark.parametrize("backend", ["triton", "auto"])
def test_packed_window_falls_back_to_per_query_when_probe_has_no_tile(
packages, monkeypatch, backend
):
from vllm_rlt.kernels.paged_attention import PackedWindowShape

monkeypatch.setattr(torch.cuda, "get_device_capability", lambda device: (8, 9))
shape = PackedWindowShape(16, 4, 36, 1024)
selected = create_backend(
backend,
torch.device("cuda"),
torch.bfloat16,
256,
16,
required_capabilities=BackendCapabilities(sliding_window=True),
packed_window_shape=shape,
)
assert selected.info["selected_backend"] == "triton"
assert selected.info["packed_prefill_tile"] is None
assert not selected.capabilities.packed_prefill
assert not selected.capabilities.packed_sliding_window


def test_auto_skips_broken_binary_import(packages, monkeypatch):
from vllm_rlt.kernels import flash_attention

Expand All @@ -326,3 +382,170 @@ def import_module(name):
assert "incompatible CUDA extension" in selected.info["selection_reason"]
with pytest.raises(OSError, match="incompatible CUDA extension"):
create_backend("flash_attn_2", torch.device("cuda"), torch.bfloat16, 128, 256)


@pytest.mark.parametrize("requested", [None, True, False])
@pytest.mark.parametrize("tile", [None, (32, 32)])
@pytest.mark.parametrize("backend", ["triton", "auto"])
def test_window_engine_resolves_tristate_before_cache_planning(
packages, monkeypatch, requested, tile, backend
):
from tests.test_dory import tiny_dory_config
from vllm_rlt.attention import backend as selection
from vllm_rlt.engine import stage_engine

monkeypatch.setattr(torch.cuda, "get_device_capability", lambda device: (8, 9))
original_import = selection.importlib.import_module
probes = []

def import_module(name):
if name == "vllm_rlt.kernels.triton_attention":

def probe(*args):
probes.append(args[-1])
return tile

return SimpleNamespace(select_packed_window_tile=probe)
return original_import(name)

monkeypatch.setattr(selection.importlib, "import_module", import_module)
config = tiny_dory_config()
model = SimpleNamespace(
config=config,
requires_boundary_kv=True,
requires_sliding_window=True,
parameters=lambda: iter(
[SimpleNamespace(device=torch.device("cuda"), dtype=torch.bfloat16)]
),
)

class Planned(Exception):
pass

def plan_cache(model, cache, scheduler, execution, selected, **kwargs):
assert execution.packed_prefill is (requested is not False and tile is not None)
assert selected == "triton"
raise Planned

monkeypatch.setattr(stage_engine, "plan_cache", plan_cache)
error = ValueError if requested is True and tile is None else Planned
with pytest.raises(error):
LLMEngine(
model,
attention_backend=backend,
execution_config=ExecutionConfig(packed_prefill=requested),
)
assert len(probes) == int(requested is not False)
if probes:
assert probes[0].query_heads == config.num_attention_heads
assert probes[0].kv_layers == config.num_kv_layers
assert probes[0].window == config.sliding_window


@pytest.mark.parametrize("missing", ["packed_prefill", "packed_sliding_window"])
@pytest.mark.parametrize("requested", [None, True])
def test_window_engine_requires_both_packed_capabilities(monkeypatch, missing, requested):
from dataclasses import replace

from tests.test_dory import tiny_dory_config
from vllm_rlt.attention import AttentionBackend
from vllm_rlt.attention import backend as selection
from vllm_rlt.engine import stage_engine

capabilities = replace(
BackendCapabilities(packed_prefill=True, sliding_window=True, packed_sliding_window=True),
**{missing: False},
)
monkeypatch.setattr(
selection,
"_create_explicit",
lambda *args, **kwargs: AttentionBackend(None, capabilities, {"backend": "triton"}),
)
model = SimpleNamespace(
config=tiny_dory_config(),
requires_boundary_kv=True,
requires_sliding_window=True,
parameters=lambda: iter(
[SimpleNamespace(device=torch.device("cuda"), dtype=torch.bfloat16)]
),
)

class Planned(Exception):
pass

def plan_cache(model, cache, scheduler, execution, selected, **kwargs):
assert requested is None
assert execution.packed_prefill is False
raise Planned

monkeypatch.setattr(stage_engine, "plan_cache", plan_cache)
error = ValueError if requested is True else Planned
with pytest.raises(error, match=missing if requested else None):
LLMEngine(
model,
attention_backend="triton",
execution_config=ExecutionConfig(packed_prefill=requested),
)


@pytest.mark.parametrize("requested", [None, True])
@pytest.mark.parametrize(
"asynchronous,graphs,preemption,role",
[
(True, False, False, "prefill_decode"),
(False, True, False, "prefill_decode"),
(False, False, True, "prefill_decode"),
(False, False, False, "decode"),
],
)
def test_incompatible_window_engine_does_not_probe_packed_tile(
packages, monkeypatch, requested, asynchronous, graphs, preemption, role
):
from tests.test_dory import tiny_dory_config
from vllm_rlt import SchedulerConfig
from vllm_rlt.attention import backend as selection
from vllm_rlt.engine import stage_engine
from vllm_rlt.engine.stage_config import StageConfig

monkeypatch.setattr(torch.cuda, "get_device_capability", lambda device: (8, 9))
original_import = selection.importlib.import_module

def import_module(name):
if name == "vllm_rlt.kernels.triton_attention":
return SimpleNamespace(
select_packed_window_tile=lambda *args: pytest.fail(
"incompatible execution probed a packed tile"
)
)
return original_import(name)

monkeypatch.setattr(selection.importlib, "import_module", import_module)
model = SimpleNamespace(
config=tiny_dory_config(),
requires_boundary_kv=True,
requires_sliding_window=True,
fixed_loop_count=True,
supports_async_scheduling=True,
parameters=lambda: iter(
[SimpleNamespace(device=torch.device("cuda"), dtype=torch.bfloat16)]
),
)

class Planned(Exception):
pass

def plan_cache(model, cache, scheduler, execution, selected, **kwargs):
assert requested is None and execution.packed_prefill is False
raise Planned

monkeypatch.setattr(stage_engine, "plan_cache", plan_cache)
with pytest.raises(ValueError if requested is True else Planned):
LLMEngine(
model,
attention_backend="triton",
stage_config=StageConfig(role=role),
scheduler_config=SchedulerConfig(enable_preemption=preemption),
execution_config=ExecutionConfig(
packed_prefill=requested, async_scheduling=asynchronous, cuda_graphs=graphs
),
)
20 changes: 20 additions & 0 deletions tests/test_dory.py
Original file line number Diff line number Diff line change
Expand Up @@ -670,3 +670,23 @@ def test_exit_written_ignores_boundary_layers(model):
# The output stage has not run, so "every layer" is still false.
assert not cache.token_written("request", 0, model.config.total_ut_steps - 1)
cache.free("request")


@pytest.mark.parametrize("packed_prefill", [None, False])
def test_dory_cpu_keeps_per_row_prefill(model, packed_prefill):
engine = LLMEngine(model, execution_config=ExecutionConfig(packed_prefill=packed_prefill))
assert engine.execution_config.packed_prefill is False
assert not engine.cache_manager.attention_capabilities.packed_sliding_window


def test_dory_packed_prefill_requires_cuda(model):
with pytest.raises(ValueError, match="packed_prefill requires CUDA"):
LLMEngine(model, execution_config=ExecutionConfig(packed_prefill=True))


def test_dory_packed_prefill_rejects_graphs(model):
with pytest.raises(ValueError, match="packed prefill requires synchronous eager"):
LLMEngine(
model,
execution_config=ExecutionConfig(packed_prefill=True, cuda_graphs=True),
)
Loading
Loading