Repository navigation
[Feature] Add Ascend Ouro support with asynchronous eager execution - #168
KrystalRay wants to merge 15 commits into
Conversation
Validation details and follow-up workAt commit Case 203 also exposed a chunking difference. Full prefill produces the final For case 203, native The earliest observed full/chunk difference was in the first-layer Q/K/V CUDA SM120 showed the opposite answer change on the same question: official HF An earlier 87-question run at commit Hardware testing used 910B3. Other Ascend variants still need device validation. The performance table uses a fixed 13.5 GiB KV pool, one excluded warmup and HTTP/SSE measurements covered 24 requests at concurrency 1 and 48 at Earlier A/B numbers compare the entire backends, whose attention arithmetic |
Merge the per-group KV-read refactor and the follow-up batching work into one reviewable change. The arithmetic contract is unchanged: BF16 score scaling, additive BF16 causal mask, FP32 softmax cast back to BF16 probabilities, then a BF16 matmul. - Build the Scatter ND write index from staged device tensors with a fail-closed row/address check. - Batch equal-length whole-prompt prefill and independent single-row decode groups; cache write slots, ragged plans and the prefill mask per batch. - Keep a single decode row on the original per-group path. - Clamp pooled-KV gather slots before index_select: plan_attention_metadata pads short block tables with -1, which CANN gather rejects. The additive mask already zeroes those columns, so clamping cannot change the output. Covered by a BF16 mixed-length ragged test. - Add CPU/NPU parity tests and record matched engine-level and full W1-W7 serving results in the Ascend recipe and README support matrix.
d921719 to
3e115df
Compare
Ascend eager batching follow-up on
|
| Workload | Input / output / C | before tok/s | merged tok/s | Δ tok/s | before TTFT p95 (s) | merged TTFT p95 (s) | before TPOT p95 (ms) | merged TPOT p95 (ms) |
|---|---|---|---|---|---|---|---|---|
| W1 | 128 / 128 / 1 | 3.944 | 4.653 | +18.0% | 0.426 | 0.205 | 265.8 | 221.1 |
| W2 | 128 / 128 / 32 | 4.802 | 96.726 | +1914.3% | 9.737 | 6.827 | 6642.6 | 285.9 |
| W3 | 4096 / 128 / 1 | 3.727 | 4.368 | +17.2% | 2.167 | 1.721 | 254.3 | 222.1 |
| W4 | 4096 / 128 / 32 | 4.201 | 24.761 | +489.4% | 747.440 | 137.225 | 1794.4 | 236.0 |
| W5 | 1024 / 512 / 8 | 4.575 | 30.744 | +572.0% | 4.676 | 2.551 | 1743.0 | 256.0 |
| W6 | ShareGPT / ≤512 / 8 | 4.660 | 21.417 | +359.6% | 3.572 | 1.689 | 1666.7 | 317.5 |
| W7 | ShareGPT / ≤512 / 32 | 4.760 | 27.970 | +487.6% | 15.765 | 7.289 | 6523.6 | 1071.7 |
All seven workloads completed three unprofiled repetitions with zero failed requests; throughput is the median of the three aggregate runs and latency percentiles pool successful requests across repetitions. Successful requests over the three runs (before→merged): W1 6→9, W2 96→192, W3 6→9, W4 96→120, W5 24→24, W6 27→33, W7 99→105.
The large W2 ratio is dominated by the baseline's lack of concurrency scaling rather than by a 20× per-request speedup: on d921719, raising concurrency from 1 to 32 moved aggregate throughput only from 3.944 to 4.802 tok/s (1.22×) while TPOT p95 grew from 265.8 ms to 6642.6 ms, whereas the merged path reaches 20.79× the C1 throughput with TPOT p95 of 285.9 ms; the same near-flat baseline scaling appears in W4, W5, W6 and W7. Under the W1–W7 protocol a faster configuration also covers more inputs in the same 60-second admission window (W2 completed 32 requests per run before versus 64 after, with drain falling from about 792 s to 25 s), so the aggregate ratio reflects both a shorter drain and a larger completed-request count.
The engine-level A/B behind this change is recorded in docs/recipes/ouro_ascend.md §5: at 128/32/8 the per-group baseline was 14.611 tok/s with 630.7 ms TTFT and 544.8 ms TPOT, versus 37.724 tok/s, 273.0 ms and 209.9 ms after batching, while official HF eager measured 50.792 tok/s, 158.9 ms and 157.5 ms in that diagnostic harness; the 2048/32/1 low-concurrency control is unchanged within noise. HF eager remains faster than the native engine in that particular engine-level comparison. These are end-to-end numbers, not kernel-count estimates.
Correctness gates
- CPU synthetic parity in BF16: regular prefill, single-request prefill, ragged decode, a mixed-length ragged batch whose short block tables carry
-1padding, single decode and an irregular grid all match the per-group path exactly (atol=0, rtol=0), with path-activation counters asserted in the new tests. - Ascend selection suite on the merged candidate (
test_ascend.py,test_ascend_eager.py,test_ascend_kv.py,test_ascend_serving.py,test_ascend_checkpoint.pywith--run-npu): 31 passed, 0 skipped, 0 failures in 66.6 s on NPU 6, including the new batched-path parity cases. - The full W1–W7 campaign completed all seven workloads with zero failed requests; the engine-level A/B reported complete generated sequences 24/24 at C8 and 3/3 at C1, and 256/256 plus 32/32 pre-lm-head vectors with
max_abs=0.0for bothmerged vs HFandmerged vs before. - The earlier full paired campaign (
kv-read-full-20261106, 73 jobs) recorded GSM8K-87 three-shot strict 63/87 for both HF eager and the target, 87/87 exact generated sequences with paired no net loss, and 8952/8952 same-history logit vectors exact withmax_abs=0.0andmax_relative_l2=0.0. Those figures qualifyd921719and a different KV-read candidate, not this change; a fresh campaign quality gate (GSM8K-87 and the paired logit vectors) on the merged candidate is still outstanding and is not claimed here. I do not treat this as blocking for the batching change because the arithmetic path is unchanged and the exact-parity gates above cover the merged candidate; I will run the campaign quality gate as a follow-up.
Remaining gaps
- Case 203 chunk128 mismatch is still unresolved (full prefill produces 31, chunk128 produces 27, first divergence at the 15th output token); this follow-up does not activate the batched prefill path under chunked prefill and makes no claim about it.
- Multi-row non-initial decode groups still use the original path. Adaptive exit, long conversations, C16+ concurrency, other checkpoints and models, async scheduling, graphs, speculation, PD and prefix caching are untested or unsupported here.
- Three timed repetitions per workload on one hardware family, with card assignment differing from the published baseline; this is not a saturation sweep and does not qualify every checkpoint, hardware or feature combination.
- Measurements, commands, versions and limitations are recorded in
docs/recipes/ouro_ascend.md§5 and reflected in the README support matrix; no standalone benchmark script is added to the repository.
Enable fixed-depth async scheduling with reusable batch metadata, owned KV in attention layout, and an optional native host plan issuing unchanged eager ATen operations. Keep the NPU runtime, batched sampling and tensor reuse in its Ascend adapter; retain the shared CUDA/ROCm model and execution paths. No NPU graph replay or fused/replacement model operators are introduced. Integrate upstream c338aaa, including sliding windows, tri-state packed prefill and the CUDA state-pool changes. Preserve the 13 existing PR commits and fold the tested Ascend engine follow-up into this one incremental commit. The code matches b6c553b except for an AST-identical Ruff formatting correction in the original Ascend KV writer. Include the CUDA test correction for automatic execution-config resolution. Validation: 87 Ascend-targeted cases, 192 fixed-input and 16 natural-input HF sequence checks, 18 unprofiled timing runs with exact HF-oracle output for 3588 native requests. Median throughput vs HF: S8 +58.92%, S64 +46.95%, L8 +26.21%; short-prompt TTFT remains worse. CUDA qualification: 1736 unique cases pass (619 GPU), 181 skipped; five paired real-weight configurations match the compatible baseline. ROCm hardware qualification remains deferred.
Keep the ordinary recurrent boundary while pending prompt or recurrent work competes under refill/mixed scheduling. Full-depth PRELUDE otherwise bypasses refill and can serialize complete generations or starve an existing request. Add bounded scheduler and single/multi-stream NPU regressions. Correct CUDA qualification wording and record all W1-W7 HF comparisons, fallback activation, latency distributions and unresolved output differences.


Enable native Ouro BF16 inference on one Ascend NPU, including an opt-in asynchronous eager execution path. Stable batches reuse metadata and owned KV in attention layout, and an optional native host executor dispatches the same individual ATen model operations.
Required human contributor answers
1. What outcome are you trying to achieve?
Enable Ouro to run through vllm-rlt’s native runtime and OpenAI-compatible
server on a single Ascend NPU. The initial scope is BF16, fixed four-loop,
synchronous greedy inference with full prefill.
2. What is the highest-leverage variable for that outcome?
End-to-end correctness of the supported NPU execution path, particularly
attention arithmetic and the causality, isolation and lifecycle of
per-depth KV state.
3. How will this work move that variable rather than create busywork?
This work adds the device/backend integration needed to execute the existing
runtime on Ascend, together with targeted attention, KV and serving regressions.
Paired real-weight checks against official HF and actual HTTP measurements
make the supported behavior and remaining gaps reviewable. Failed checks,
including the chunk128 sequence mismatch, remain documented without relaxing
the acceptance threshold.
4. Who is the customer or user, and what do they actually need?
Ouro users with Ascend hardware need a reproducible native deployment path,
documented operating limits, correct KV behavior, and measured latency and
memory costs.
Maintainers need an optional integration with clear boundaries and repeatable
regression tests so they can assess and maintain this hardware path.
PR necessity and bugfix evidence
Yes. A PR is needed to add NPU device dispatch, attention-backend integration and KV support to the native engine and serving entrypoints.
Classification: Feature / experimental hardware enablement. The original synchronous NPU path is extended with asynchronous execution and opt-in execution/cache capabilities. The ancillary CUDA test correction below addresses an observed assertion failure reproduced on both the compatible baseline and this implementation.
Purpose
The measured Ascend bottleneck is host submission and repeated metadata/KV preparation. This change adds:
ascend_eagerattention backend.[batch, kv_head, token, head_dim]layout; generic accesses export live histories to paged storage.native_eager_planhost executor that releases the Python GIL and issues individual eager ATen operations. Q/K/V and gate/up projections remain separate, with explicit RMSNorm, RoPE, matmul and softmax. It borrows live weights and uses Python for unsupported module schemas/hooks.NPU runtime adaptation, batched sampling and tensor reuse live in
worker/ascend.py. The shared model, ModelRuntime, Sampler and Workspace implementations remain at the upstream-compatible baseline. CUDA stream dispatch resolves to the originaltorch.cudamodule. New backend capability fields preserve existing positional arguments.Submitted-write coverage is distinct from device completion: events still protect readers, cancellation, request-ID reuse and release.
The new cache/native-executor options default off. The Python executor remains available without a compiler; the throughput claim below applies to the combined configuration including native host dispatch, rather than async scheduling alone. Native dispatch needs a C++17 compiler, Ninja and PyTorch headers on first enabled startup. No NPU Graph replay or fused/replacement model operators are introduced.
Setup, startup and request reproduction are in the Ascend recipe. A fixed-UT4 CLI example is:
Follow-up: refill progress and full W1–W7 comparison
Incremental commit:
394f3b18e90dc3f3c0365cbd3e4bfbe074deb0dc, directly after028f9c6.With a 1,024-token budget and P1024 prompts, full-depth decode PRELUDE could cycle directly back to CODA and prevent other prompts from being prefetched. A first attempt to yield only to prompts could then starve an existing RECURRENT request. The final eligibility guard preserves that boundary while WAITING, runnable prompt or RECURRENT work competes under refill/mixed scheduling. Stable batches still coalesce. The guard is short-circuited when the NPU-only option is disabled; CUDA/ROCm execution remains unchanged.
The increment changes only
core/scheduler.py, its existing fixed-prelude regression file, and the Ascend recipe. Bounded CPU tests reproduce both starvation cases before the fix; single/multi-stream NPU tests check interleaved progress, complete output parity and KV release.Validation: full CPU 1121 passed / 802 skipped, 51 targeted CPU/NPU cases passed, and full-tree Ruff lint/format passed. The recipe now records the completed CUDA hardware validation instead of transient server reachability. ROCm hardware validation remains deferred.
Full applicable single-stage protocol from
docs/unified_execution.md: all seven workloads, native combination versus official HF eager, three unprofiled 60-second closed-loop admission windows per arm followed by complete drain (42 selected runs). Both use BF16/UT4/greedy, fixed device/CPU affinity per case and TASK_QUEUE_ENABLE=2. The engine uses refill with budget/chunk 1024. W1–W5 ignore EOS; W6/W7 replay the frozen 256-input ShareGPT selection with natural EOS and cap 512. These are in-process actor timings; CUDA-only FA4/graphs/PD/speculation modes are outside this Ascend comparison.64 GiB capacity adaptation is applied to both arms: W4 retains 32 clients with four resident requests; W6/W7 allow at most eight residents. Other resident limits equal client concurrency.
All 42 selected runs complete with 1087 successful requests and zero runtime failures. Complete paired-output matches for W1–W7 are 10/10, 221/224, 9/9, 93/108, 18/24, 15/27 and 45/99. Three supplemental fixed-input HF checks after timing establish 15/15 W1 and 13/13 W3 matches across all native requests. A bounded natural-input check also observes different HF outputs at batch 1 and batch 8; it does not explain all cross-engine differences. Full greedy equivalence and equal-quality acceptance have not passed. Natural mean output lengths differ (HF/native): W6 458.33/452.44, W7 457.15/466.58 tokens.
Native executor layer calls are active without fallback throughout W1/W3. W2/W4/W5/W6/W7 materialize paged KV during warmup and have zero native-executor layer calls in the timed windows; their layer work uses the Python eager fallback. W7’s +1.14% median difference is within the observed run ranges, and W3 has substantial native variability, including one slower paired run. Pooled mean/p95/p99 TTFT, TPOT, ITL and E2E are recorded in recipe §7; these are case-dependent serving results, not isolated operator or async-only gains.
These results compare the whole enabled combination, with fallback activation reported. They do not isolate async scheduling or the native executor. Case-dependent throughput and unresolved output differences do not support a universal or equal-quality speedup claim. The earlier tuned O32 table remains historical evidence with different budgets/chunking.
Original integration validation (
028f9c6)vllm-rlt Version / Commit:
028f9c6287d9d2ff63c9f848ae05b678553416fcThe original integration commit added one increment after
3e115df, preserving the original 13 PR commits and integrating upstreamc338aaa. The resulting tree matches testedb6c553bapart from one AST-identical formatting correction in the existing Ascend KV writer; its affected NPU tests are re-run. Upstream sliding windows, automatic packed-prefill policy and CUDA state-pool changes are retained.pre-commit run --all-files: Ruff 0.13.2 lint/format and the CPU regression hook, on the exact proposed tree.bd257388).TASK_QUEUE_ENABLE=2, one execution stream and OMP/MKL threads set to one before import. Startup, compilation and a complete shape-matched warmup are excluded. Each run admits closed-loop traffic for 60 seconds and then fully drains; throughput includes the drain. Three repetitions alternate native/HF, HF/native, native/HF.Prior tuned O32 results (
028f9c6)Ascend 910B3, CANN 9.0.0, Torch/Torch-NPU 2.10, Transformers 4.54.1, Ouro-1.4B BF16, UT4, greedy output. Official HF model source is pinned to
574fa66cb8bf5abdc979642d01cf2b79b16bfab1, using a public DynamicCache adapter for Transformers 4.54.1 compatibility. Weights SHA256:58872a72616c736595b8b7662079c5b12c5a162ec16eae94f21c348dfa9885af. Timing uses complete equal-length prefill and forces 32 output tokens with EOS ignored. Figures are median output tok/s; brackets show the three-run min–max range.Token budget / chunk / reserved blocks are respectively 1024/1024/336, 8192/1024/2576 and 16384/4096/4144, with 16-token blocks. Native exceeds HF in each of the nine paired comparisons. These are engine-actor measurements, not HTTP latency.
Throughput trades off short-prompt first-output latency: pooled TTFT p50 is 222.4 vs 157.5 ms at C8 and 634.5 vs 595.5 ms at C64 (native/HF). Long-prompt TTFT including queuing is 11.68 vs 13.54 s. Peak allocated memory in GiB (native/HF) is 3.616/3.610, 10.209/10.167 and 25.087/25.243.
CUDA compatibility
On RTX PRO 6000 SM120 with Torch 2.11/CUDA 12.8, staged/resumed validation covers 1736 unique passing cases, including 619 GPU cases, with 181 skipped. This is not a single uninterrupted zero-failure suite: unavailable FlashAttention/NIXL cases were classified explicitly, and one 18 MiB allocator-budget variance passed subsequent reruns; both baseline and candidate pass the full 69-case CDB file.
The eight dynamic-async cases originally failed on both versions because an assertion compared resolved graph-row/packed-prefill defaults with unresolved inputs. The test-only correction preserves all other execution-switch, state-pool and output-parity assertions.
Real Ouro weights match the compatible baseline in Torch sync, Triton sync, async single-stream, async multi-stream and async graph modes for 128/4096/41-token inputs with mixed exit depths. All 15 paired requests match token IDs, exit depths and finish reasons; graph mode captures three entries and replays 39 times with zero fallback. CUDA throughput was not remeasured.
Limits and remaining qualification