Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 39 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

[![Python 3.10+](https://img.shields.io/badge/Python-3.10%2B-3776AB?style=flat-square)](pyproject.toml)
[![CLI](https://img.shields.io/badge/Interface-CLI-14B8A6?style=flat-square)](docs/quickstart.md)
[![vLLM-Omni PR](https://img.shields.io/badge/vLLM--Omni-Upstream_PR-8B5CF6?style=flat-square)](https://github.com/vllm-project/vllm-omni/pull/8739)
[![vLLM-Omni contributions](https://img.shields.io/badge/vLLM--Omni-Contributions-8B5CF6?style=flat-square)](#results)

[Why InferRSI](#why-inferrsi) · [How It Works](#how-it-works) · [Results](#results) · [Quickstart](#quickstart) · [Architecture](#architecture) · [Docs](#docs)

Expand Down Expand Up @@ -51,7 +51,9 @@ The **offline harness-improvement MVP** adds a separate Skill Eval path for supp
declarative candidates. Its fixed-output experiments validate the controller;
real-model gains remain unmeasured. [Flow, limits and reproduction →](docs/harness-improvement.md)

## Choose your starting path
<a id="choose-your-starting-path"></a>

## 🧭 Choose your starting path

| You want to… | Start at |
| --- | --- |
Expand All @@ -63,30 +65,47 @@ For every CLI/Python operation — discovery, normalized reads, store candidates

<a id="results"></a>

## 📊 From profiling to a 30.60% latency reduction

**Qwen-Image-2.1-Turbo · 4× NVIDIA B300 · 1280×704 · BF16 · Native 8 steps**

| Baseline E2E | Optimized E2E | Image validation |
| :---: | :---: | :---: |
| **384.16 ms** | **266.60 ms** | **44/44 pixel-identical** |
## 📊 InferRSI × vLLM-Omni: Optimization Results

Recorded local campaign: five alternating AB/BA groups, warmed repeated prompts with new seeds, and complete RGBA PNG responses. Accuracy checks cover 40 formal images and 4 held-out images. Latencies are medians of trial means for this workload.
InferRSI campaigns and upstream experiments cover image, video and speech models.
Each result applies to its linked workload and measured revisions. Some results
are reported directly by upstream contributors, with validation scoped to their benchmarks.

[Results, source pins & every iteration →](docs/qwen-image-2.1-turbo-b300.md) · [Acceptance issue](https://github.com/InferMatrixLab/InferRSI/issues/3) · [Upstream PR](https://github.com/vllm-project/vllm-omni/pull/8739)
<details open>
<summary>Benchmarks, validation limits, and upstream PRs</summary>

### Ascend NPU: up to 22.0% reported latency reduction

**Qwen-Image-2.1-Turbo · 1× Ascend 910B1 · 1280×704 · BF16 · Native 8 steps · eager**
| Model | Hardware | Optimization approach | Measured gains | Validation and limits | Evidence | vLLM-Omni PR |
| --- | --- | --- | --- | --- | --- | --- |
| **Qwen-Image-2.1-Turbo** | 4× B300 | <ul><li>Text-conditioning reuse</li><li>BF16 operator fusion</li><li>Symmetric-memory Ulysses with batched QKV exchange</li></ul> | **384.16 → 266.60 ms (−30.60%)**<br>Warm HTTP end-to-end latency, repeated prompts | Historical InferRSI campaign. Latencies are medians of five trial means from alternating AB/BA groups. All 44 RGBA images were pixel-identical. | [Campaign #3](https://github.com/InferMatrixLab/InferRSI/issues/3) · [Pinned results](docs/qwen-image-2.1-turbo-b300.md) | [#8739](https://github.com/vllm-project/vllm-omni/pull/8739) (merged) |
| **Qwen-Image-2.1-Turbo** | 1× Ascend 910B1 | <ul><li>Conditioning cache for repeated prompts</li><li>Shared modulation once per denoising step</li><li>Native interleaved NPU RoPE with frequency-table reuse</li></ul> | **−22.20%** offline generation latency<br>Warm repeated prompts | Upstream parent-to-merge comparison. 18/18 RGBA images were pixel-identical. The 30% target was unmet. Both arms aborted during teardown, leaving lifecycle `INCONCLUSIVE`. | [Final-merge measurements](https://github.com/vllm-project/vllm-omni/pull/8729#:~:text=Final%2Dmerge%20performance%20and%20image%20comparison) · [Earlier Ascend cohorts](docs/qwen-image-2.1-turbo-ascend.md) | [#8729](https://github.com/vllm-project/vllm-omni/pull/8729) (merged) |
| **MiniMax-H3 Ref2VA** | 8× Ascend 910B1 | <ul><li>Lossy Cache-DiT profile tuning</li></ul> | **407.27 → 108.12 s (−73.45%)**<br>5 s video, HTTP end-to-end vs. dense inference | All 14 frozen quality comparisons passed. | [Campaign #293](https://github.com/InferMatrixLab/InferRSI/issues/293) · [Pinned results](evidence/h3-ref2va-ascend-293/results/final-5s/README.md) | [#9698](https://github.com/vllm-project/vllm-omni/pull/9698) (draft) |
| **MiniMax-H3 Ref2VA** | 4 logical Ascend 910C devices | <ul><li>Ref2VA migration changes</li><li>Opt-in BF16 prefix attention</li></ul> | **291.25 → 286.67 s (−1.57%)**<br>5 s video, HTTP end-to-end | One lossless screening pair. Strict quality checks passed for the screened candidate. Overall campaign `INCONCLUSIVE`. | [Campaign #297](https://github.com/InferMatrixLab/InferRSI/issues/297) · [Evidence PR #317](https://github.com/InferMatrixLab/InferRSI/pull/317) | [#9745](https://github.com/vllm-project/vllm-omni/pull/9745) (draft) |
| **MiniMax-H3-Image** | 4× Ascend 910B1 | <ul><li>Opt-in VAE tile graphs</li><li>Approximate encoder/decoder tile batching</li></ul> | **16.89 → 10.82 s** · 4736×3552 generation<br>**4.18 → 2.67 s** · 1024×1024 generation<br>**5.68 → 4.57 s** · 1024×1024 editing | Single-launch HTTP medians vs. the same-layout SP4 control. All 30 numerical image screens passed. Latency targets were unmet and final qualification remains `INCONCLUSIVE`. | [Campaign #390](https://github.com/InferMatrixLab/InferRSI/issues/390) · [Evidence PR #394](https://github.com/InferMatrixLab/InferRSI/pull/394) | [#10184](https://github.com/vllm-project/vllm-omni/pull/10184) (open) |
| **MiniMax-H3-Image** | 2× RTX 3090 | <ul><li>Rank-local layerwise offload</li><li>Selected DiT weights kept resident</li><li>AdaLN caching</li><li>Request-local text refinement reuse</li></ul> | **36.62 → 14.59 s (−60.16%)**<br>Warm mean HTTP delivery latency | One upstream A/B of complete deployment profiles. All nine image pairs had identical decoded RGB pixels, including warmups. Repeated qualification remains incomplete. | [Paired deployment benchmark](https://github.com/vllm-project/vllm-omni/pull/10243#:~:text=One%20paired%20deployment%20comparison) | [#10243](https://github.com/vllm-project/vllm-omni/pull/10243) (open) |
| **MiniCPM-o 4.5** | 1× L20X | <ul><li>Packed conditioning on a tuned configuration</li><li>TensorRT speaker extraction + CUDA MPS (separate comparison)</li></ul> | **+15.55%** throughput · **−27.69%** median TTFA<br>Packed conditioning, C32, same 50-frame window<br>**+20.5%** at C16 · **+25.4%** at C32<br>TensorRT + MPS throughput | The packed-conditioning candidate missed the exploratory WER margin against the tuning reference at the 50-frame setting. The separate TensorRT + MPS comparison had speech-quality verdict `FAIL`. | [Campaign #49](https://github.com/InferMatrixLab/InferRSI/issues/49) | [#9218](https://github.com/vllm-project/vllm-omni/pull/9218), [#9188](https://github.com/vllm-project/vllm-omni/pull/9188) (both merged) |
| **Qwen3-Omni** | 2× L20X | <ul><li>cuDNN codec autotuning in the MRv2 deployment</li></ul> | **+7.79%** throughput · **−6.06%** mean RTF<br>One C64 pilot | Median time to first audio **regressed by 7.12%**. Campaign `INCONCLUSIVE`. | [Campaign #112](https://github.com/InferMatrixLab/InferRSI/issues/112) · [Pinned pilot](https://github.com/InferMatrixLab/InferRSI/tree/e1ae256fc075ea9ab64aec68da749cf4a8c22ac7/evidence/qwen3-omni-2gpu-20261023/results/pr9283-cudnn-performance) | [#9283](https://github.com/vllm-project/vllm-omni/pull/9283) (draft) |
| **LFM3-Omni-2B** | 1× Ascend 910B1 | <ul><li>Depth graphs</li><li>Grouped greedy sampling</li></ul> | **0.285 → 0.593 requests/s**<br>Upstream offline Omni API A/B, C4, temperature 0 | Retained audio waveforms matched bit-for-bit within each tested temperature. | [Offline A/B benchmark](https://github.com/vllm-project/vllm-omni/pull/9827#:~:text=Performance%20results) | [#9827](https://github.com/vllm-project/vllm-omni/pull/9827) (merged) |
| **LingBot-World** | 4× Ascend 910C1 | <ul><li>Whole-block paged KV gather with <code>index_select</code></li></ul> | **108.29 → 103.89 s (−4.06%)**<br>Full-request latency, upstream AB/BA | Pre-codec tensors and decoded RGB frames matched exactly. | [KV gather benchmark](https://github.com/vllm-project/vllm-omni/pull/8112#:~:text=Performance%3A%20whole%2Dblock%20index_select%20KV%20gather%20on%20NPU) | [#8112](https://github.com/vllm-project/vllm-omni/pull/8112) (merged) |

**Earlier Qwen-Image-2.1-Turbo Ascend measurements.** On one Ascend 910B1 at
1280×704, BF16, eager, and eight native steps, the author reported a 3111.5 ms
baseline. Conditioning reuse with interleaved RoPE reached **2459.6 ms (−21.0%)**
with pixel-identical RGBA. Adding the opt-in VAE fastpath reached **2425.5 ms
(−22.0%)** and passed the reported quality gate with changed pixels. Each mean
covers three requests after two warmups. InferRSI verification covered the
import, so acceptance remains `INCONCLUSIVE`. The 30% target was unmet. The B300
campaign and this cohort used different hardware and measurement protocols.
[Earlier Ascend report](docs/qwen-image-2.1-turbo-ascend.md)

TTFA is time to first audio. RTF is real-time factor. WER is word error rate.

Upstream PR statuses checked on 2026-11-12.

| Optimization | Mean vs 3111.5 ms baseline | Reported image validation |
| --- | :---: | --- |
| Conditioning reuse + interleaved RoPE | **2459.6 ms (−21.0%)** | Pixel-identical RGBA |
| + opt-in VAE fastpath | **2425.5 ms (−22.0%)** | Quality gate passed; not pixel-identical |
</details>

Author-reported means from three requests after two warmups; not independently verified by this import. **The 30% target is not met.** The B300 and NPU cohorts use separate hardware and measurement protocols.
<a id="standalone-engine-benchmarks"></a>

[Ascend measurements and rejected rounds →](docs/qwen-image-2.1-turbo-ascend.md) · [Upstream NPU patch](https://github.com/vllm-project/vllm-omni/pull/8729)
## 🧪 Standalone engine benchmarks

### Standalone video/audio engines

Expand Down
Loading