Skip to content

[perf][dory] add opt-in packed Triton prefill - #189

Merged
bjf-frz merged 7 commits into
InferMatrixLab:mainfrom
cecilsd:perf/dory-packed-prefill
Nov 17, 2026
Merged

bjf-frz merged 7 commits into
InferMatrixLab:mainfrom
cecilsd:perf/dory-packed-prefill

Conversation

@cecilsd

@cecilsd cecilsd commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Required human contributor answers

1. What outcome are you trying to achieve?

Reduce Dory's long-prompt time to first token.

2. What is the highest-leverage variable for that outcome?

The cost of running attention over long prompts during prefill.

3. How will this work move that variable rather than create busywork?

Use an opt-in Triton kernel that tiles queries and reuses KV reads. Keep per-query attention for batches containing a short prompt.

4. Who is the customer or user, and what do they actually need?

Dory users serving long prompts on RTX 5090 who need lower time to first token and clear hardware and BF16 reproducibility limits.

PR necessity and bugfix evidence

A PR is needed to expose and validate the packed full and sliding-window attention path for Dory. This is an opt-in performance feature.

Purpose

  • Add ExecutionConfig(dory_packed_prefill=True) and --dory-packed-prefill. The default is per-query attention.
  • Use 32 × 64 tiles, one pipeline stage, and a residual term for BF16 rounding of the softmax weights. The option requires Dory BF16, NVIDIA SM 12.0, per-loop KV, and ordinary synchronous eager prefill.
  • Use packed attention in PRELUDE, the recurrent loops, and prompt-history CODA when every prompt in the batch is longer than 64 tokens. Any prompt of 64 tokens or less keeps the batch on per-query attention. First-token sampling and decode use per-query attention.

Hardware coverage

GPU validation covers one RTX 5090 only. Other SM 12.0 devices still need device-specific correctness and performance validation. Other GPU architectures require their own Triton tile and pipeline configuration before enabling this path. The 5090 kernel configuration stays within its per-block shared-memory limit for Dory's 256-dimensional heads.

BF16 output alignment

  • Same batch shapes and token histories. Both paths use the same causal and sliding-window ranges. The packed kernel changes the floating-point reduction order and rounds intermediate softmax weights for the matrix product with V. The compensation reduces this rounding error. Residual differences can propagate through Dory's recurrent layers. The 8K and 16K controlled runs matched all 144 short-request logit rows exactly with the per-query fallback. Long-request logits can still differ.
  • Natural request arrivals. A faster long prefill can change which requests share a batch, affecting BF16 even on the default path. A controlled 16K replay using only per-query attention reproduced the same two short-request output differences when the admission schedule changed. The timed runs matched 12/12 long-request sequences and 34/36 short-request sequences. For strict reproduction, use per-query attention and fix the batch shapes.

Test Plan

vllm-rlt Version / Commit
Base d090670, head b17d1f6.

  • Linux pre-commit run --all-files and full CPU regression suite.
  • RTX 5090 pytest -q --run-gpu tests/test_dory_gpu.py tests/test_attention.py -k 'not flash'. Cover full and sliding-window attention, skipping an entire key tile, compensated multiplication, and short-prompt routing in single-stream and default multi-stream execution.
  • Real Dory 4B BF16 checkpoint on one RTX 5090, PyTorch 2.13.0+cu130, Triton 3.7.1. Compare the same engine with the option off and on. Run three interleaved pairs after warmup. Use two loops, 256-token prefill chunks, and fixed output lengths of 8 long-request tokens and 12 short-request tokens. The mixed load adds six 32-token requests while the long prompt is in prefill.

Test Result

  • All Linux pre-commit hooks passed. The standalone full Linux CPU suite recorded 1,047 passed, 744 skipped. The targeted GPU suite recorded 132 passed on RTX 5090.
  • Windows pre-commit lint and formatting passed. Two unrelated Windows CPU failures, in Huginn dense-reference parity and PD worker-exit handling, also reproduced on the base commit.
Workload and metric Per query Packed with short-prompt fallback
8K single-request median TTFT 4.28 s 1.85 s
16K single-request median TTFT 12.02 s 3.86 s
8K mixed median batch completion 5.91 s 3.70 s
16K mixed median batch completion 13.54 s 5.65 s
8K mixed short-request TTFT p95 284 ms 274 ms
16K mixed short-request TTFT p95 284 ms 248 ms

Each mixed p95 is based on 18 short requests. Separate operator-only and full profiles of a warmed 256-token prefill step at 4K context observed 112 attention kernel launches with per-query attention and 56 with packed attention. The operator-only trace measured 92.7 ms and 9.25 ms of summed attention-kernel device time, respectively.

Comment thread vllm_rlt/entrypoints/runtime_args.py Outdated
help="Keep per-row prefill even when packed attention is available",
)
packed_prefill_mode.add_argument(
"--dory-packed-prefill",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please reuse the existing --packed-prefill option instead of adding --dory-packed-prefill. Extend the shared eligibility checks and backend capability selection to support packed sliding-window attention, keeping hardware restrictions in the backend and avoiding model-specific checks in StageEngine.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in ffca2ec and a5ea294. Dory now reuses --packed-prefill, and the shared eligibility checks cover packed sliding-window prefill. StageEngine passes the attention geometry to backend selection, and the Triton backend checks hardware support and shared-memory usage at startup. Per-query attention is the default and the fallback if the check fails.

Both GPUs used the same Dory 4B BF16 checkpoint and inputs. All weights fit in VRAM on the RTX 5090. Fully resident initialization ran out of GPU memory on the RTX 4060, so the benchmark script kept about 2.97 GiB of BF16 weights on the CPU in both runs and copied them to the GPU as needed. The timings include those transfers.

The table reports warmed engine-side medians over three repeats per configuration on the RTX 5090 and two on the RTX 4060. The mixed workload used one 4K prompt and three short prompts of 32, 32, and 64 tokens, with arrivals fixed by scheduler step.

Metric, per-query → 32 × 32 RTX 5090, all weights in VRAM RTX 4060, CPU weight staging
2K single-request time to first token 707 → 455 ms 8.30 → 5.33 s
8K single-request time to first token 4.20 → 1.82 s 42.19 → 22.97 s
8K single-request median inter-token latency 54.3 → 53.5 ms 532 → 537 ms
Mixed workload, long-request time to first token 2.29 → 1.56 s 24.90 → 18.22 s
Mixed workload, short-request p95 inter-token latency 314 → 227 ms 4.03 → 2.84 s
Mixed workload, output throughput 11.46 → 15.75 tokens/s 1.08 → 1.40 tokens/s

Greedy token IDs matched across all 14 paired request cases, with eight generated tokens per request. For Q/K/V samples from the model’s forward pass, both paths had an RMSE against the FP32 reference of about 1.7 × 10⁻⁵ for full attention and 3.1 × 10⁻⁵ for sliding-window attention. The largest observed difference in an output token’s probability was 2.95 percentage points.

32 × 32 is a common configuration validated on these two GPUs. The shared-memory check determines whether the tile fits. The fastest tile can vary by GPU and workload, so choosing it on other GPUs requires measurements with representative prompt lengths, batch shapes, and sliding-window settings.

@bjf-frz bjf-frz left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please preserve the tri-state contract: None automatically enables packed prefill when supported; explicit True must fail if unsupported; False disables it.

The rest of the implementation looks good to me.

Comment thread vllm_rlt/engine/stage_engine.py Outdated
cache_config.block_size,
required_capabilities=BackendCapabilities(
packed_prefill=bool(self.execution_config.packed_prefill),
packed_prefill=require_packed and not sliding_window,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For an explicit sliding-window packed-prefill request, require both packed_prefill and packed_sliding_window. Reuse the existing backend capability validation to reject unsupported configurations.

Comment thread vllm_rlt/engine/packed_prefill.py Outdated
raise ValueError("packed_prefill requires CUDA packed prefill with last_exited KV")
if requested is not None:
return requested
return (

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove the unconditional sliding-window exclusion from automatic selection. Check the actual execution constraints: return False for incompatible automatic configurations and raise for incompatible explicit requests.

Comment thread vllm_rlt/engine/stage_engine.py Outdated
PackedWindowShape(
config.num_attention_heads, config.num_key_value_heads, kv_layers, window
)
if require_packed and sliding_window

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probe packed sliding-window support whenever eligible, including automatic mode, rather than only for explicit requests. Also resolve packed_prefill only when it is None; the condition below currently allows explicit True to be silently overwritten with False.

@cecilsd cecilsd Nov 17, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated as suggested to preserve the tri-state semantics of packed_prefill:

  • None: automatically enable when execution constraints and backend capabilities support it, including sliding-window attention.
  • True: fail clearly when unsupported, without silently falling back.
  • False: keep packing disabled and do not probe the packed sliding-window tile.

Explicit sliding-window requests now require both packed_prefill and packed_sliding_window, using the existing backend capability validation. Eligible automatic configurations also probe the tile, and the configuration is resolved only when packed_prefill is None.

Added regression tests for missing capabilities, failed probes and incompatible execution modes, and updated the documentation and CLI help.

@bjf-frz bjf-frz changed the title perf(dory): add opt-in packed Triton prefill [perf][dory] add opt-in packed Triton prefill Nov 17, 2026
@bjf-frz

bjf-frz commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

@cecilsd Hi, did you join the wechat group? You can scan the code in readme and discussed directly.

@bjf-frz

bjf-frz commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

lgtm

@bjf-frz
bjf-frz merged commit 33d950d into InferMatrixLab:main Nov 17, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants