Repository navigation
[perf][dory] add opt-in packed Triton prefill - #189
Conversation
| help="Keep per-row prefill even when packed attention is available", | ||
| ) | ||
| packed_prefill_mode.add_argument( | ||
| "--dory-packed-prefill", |
There was a problem hiding this comment.
Please reuse the existing --packed-prefill option instead of adding --dory-packed-prefill. Extend the shared eligibility checks and backend capability selection to support packed sliding-window attention, keeping hardware restrictions in the backend and avoiding model-specific checks in StageEngine.
There was a problem hiding this comment.
Addressed in ffca2ec and a5ea294. Dory now reuses --packed-prefill, and the shared eligibility checks cover packed sliding-window prefill. StageEngine passes the attention geometry to backend selection, and the Triton backend checks hardware support and shared-memory usage at startup. Per-query attention is the default and the fallback if the check fails.
Both GPUs used the same Dory 4B BF16 checkpoint and inputs. All weights fit in VRAM on the RTX 5090. Fully resident initialization ran out of GPU memory on the RTX 4060, so the benchmark script kept about 2.97 GiB of BF16 weights on the CPU in both runs and copied them to the GPU as needed. The timings include those transfers.
The table reports warmed engine-side medians over three repeats per configuration on the RTX 5090 and two on the RTX 4060. The mixed workload used one 4K prompt and three short prompts of 32, 32, and 64 tokens, with arrivals fixed by scheduler step.
Metric, per-query → 32 × 32 |
RTX 5090, all weights in VRAM | RTX 4060, CPU weight staging |
|---|---|---|
| 2K single-request time to first token | 707 → 455 ms | 8.30 → 5.33 s |
| 8K single-request time to first token | 4.20 → 1.82 s | 42.19 → 22.97 s |
| 8K single-request median inter-token latency | 54.3 → 53.5 ms | 532 → 537 ms |
| Mixed workload, long-request time to first token | 2.29 → 1.56 s | 24.90 → 18.22 s |
| Mixed workload, short-request p95 inter-token latency | 314 → 227 ms | 4.03 → 2.84 s |
| Mixed workload, output throughput | 11.46 → 15.75 tokens/s | 1.08 → 1.40 tokens/s |
Greedy token IDs matched across all 14 paired request cases, with eight generated tokens per request. For Q/K/V samples from the model’s forward pass, both paths had an RMSE against the FP32 reference of about 1.7 × 10⁻⁵ for full attention and 3.1 × 10⁻⁵ for sliding-window attention. The largest observed difference in an output token’s probability was 2.95 percentage points.
32 × 32 is a common configuration validated on these two GPUs. The shared-memory check determines whether the tile fits. The fastest tile can vary by GPU and workload, so choosing it on other GPUs requires measurements with representative prompt lengths, batch shapes, and sliding-window settings.
bjf-frz
left a comment
There was a problem hiding this comment.
Please preserve the tri-state contract: None automatically enables packed prefill when supported; explicit True must fail if unsupported; False disables it.
The rest of the implementation looks good to me.
| cache_config.block_size, | ||
| required_capabilities=BackendCapabilities( | ||
| packed_prefill=bool(self.execution_config.packed_prefill), | ||
| packed_prefill=require_packed and not sliding_window, |
There was a problem hiding this comment.
For an explicit sliding-window packed-prefill request, require both packed_prefill and packed_sliding_window. Reuse the existing backend capability validation to reject unsupported configurations.
| raise ValueError("packed_prefill requires CUDA packed prefill with last_exited KV") | ||
| if requested is not None: | ||
| return requested | ||
| return ( |
There was a problem hiding this comment.
Please remove the unconditional sliding-window exclusion from automatic selection. Check the actual execution constraints: return False for incompatible automatic configurations and raise for incompatible explicit requests.
| PackedWindowShape( | ||
| config.num_attention_heads, config.num_key_value_heads, kv_layers, window | ||
| ) | ||
| if require_packed and sliding_window |
There was a problem hiding this comment.
Probe packed sliding-window support whenever eligible, including automatic mode, rather than only for explicit requests. Also resolve packed_prefill only when it is None; the condition below currently allows explicit True to be silently overwritten with False.
There was a problem hiding this comment.
Updated as suggested to preserve the tri-state semantics of packed_prefill:
None: automatically enable when execution constraints and backend capabilities support it, including sliding-window attention.True: fail clearly when unsupported, without silently falling back.False: keep packing disabled and do not probe the packed sliding-window tile.
Explicit sliding-window requests now require both packed_prefill and packed_sliding_window, using the existing backend capability validation. Eligible automatic configurations also probe the tile, and the configuration is resolved only when packed_prefill is None.
Added regression tests for missing capabilities, failed probes and incompatible execution modes, and updated the documentation and CLI help.
|
@cecilsd Hi, did you join the wechat group? You can scan the code in readme and discussed directly. |
|
lgtm |
Required human contributor answers
1. What outcome are you trying to achieve?
Reduce Dory's long-prompt time to first token.
2. What is the highest-leverage variable for that outcome?
The cost of running attention over long prompts during prefill.
3. How will this work move that variable rather than create busywork?
Use an opt-in Triton kernel that tiles queries and reuses KV reads. Keep per-query attention for batches containing a short prompt.
4. Who is the customer or user, and what do they actually need?
Dory users serving long prompts on RTX 5090 who need lower time to first token and clear hardware and BF16 reproducibility limits.
PR necessity and bugfix evidence
A PR is needed to expose and validate the packed full and sliding-window attention path for Dory. This is an opt-in performance feature.
Purpose
ExecutionConfig(dory_packed_prefill=True)and--dory-packed-prefill. The default is per-query attention.Hardware coverage
GPU validation covers one RTX 5090 only. Other SM 12.0 devices still need device-specific correctness and performance validation. Other GPU architectures require their own Triton tile and pipeline configuration before enabling this path. The 5090 kernel configuration stays within its per-block shared-memory limit for Dory's 256-dimensional heads.
BF16 output alignment
Test Plan
vllm-rlt Version / Commit
Base
d090670, headb17d1f6.pre-commit run --all-filesand full CPU regression suite.pytest -q --run-gpu tests/test_dory_gpu.py tests/test_attention.py -k 'not flash'. Cover full and sliding-window attention, skipping an entire key tile, compensated multiplication, and short-prompt routing in single-stream and default multi-stream execution.Test Result
Each mixed p95 is based on 18 short requests. Separate operator-only and full profiles of a warmed 256-token prefill step at 4K context observed 112 attention kernel launches with per-query attention and 56 with packed attention. The operator-only trace measured 92.7 ms and 9.25 ms of summed attention-kernel device time, respectively.