From a8bfeb629212544dde23a2a9ec85023b7440b787 Mon Sep 17 00:00:00 2001 From: zengchuang Date: Thu, 12 Nov 2026 14:51:50 +0800 Subject: [PATCH 01/11] docs: showcase InferRSI and vLLM-Omni optimization results Agent-assisted-by: Codex (author) Signed-off-by: zengchuang --- README.md | 46 +++++++++++++++++++++------------------------- 1 file changed, 21 insertions(+), 25 deletions(-) diff --git a/README.md b/README.md index 6f1893956..65baea1c2 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ [![Python 3.10+](https://img.shields.io/badge/Python-3.10%2B-3776AB?style=flat-square)](pyproject.toml) [![CLI](https://img.shields.io/badge/Interface-CLI-14B8A6?style=flat-square)](docs/quickstart.md) -[![vLLM-Omni PR](https://img.shields.io/badge/vLLM--Omni-Upstream_PR-8B5CF6?style=flat-square)](https://github.com/vllm-project/vllm-omni/pull/8739) +[![vLLM-Omni contributions](https://img.shields.io/badge/vLLM--Omni-Contributions-8B5CF6?style=flat-square)](#results) [Why InferRSI](#why-inferrsi) · [How It Works](#how-it-works) · [Results](#results) · [Quickstart](#quickstart) · [Architecture](#architecture) · [Docs](#docs) @@ -63,30 +63,26 @@ For every CLI/Python operation — discovery, normalized reads, store candidates -## 📊 From profiling to a 30.60% latency reduction - -**Qwen-Image-2.1-Turbo · 4× NVIDIA B300 · 1280×704 · BF16 · Native 8 steps** - -| Baseline E2E | Optimized E2E | Image validation | -| :---: | :---: | :---: | -| **384.16 ms** | **266.60 ms** | **44/44 pixel-identical** | - -Recorded local campaign: five alternating AB/BA groups, warmed repeated prompts with new seeds, and complete RGBA PNG responses. Accuracy checks cover 40 formal images and 4 held-out images. Latencies are medians of trial means for this workload. - -[Results, source pins & every iteration →](docs/qwen-image-2.1-turbo-b300.md) · [Acceptance issue](https://github.com/InferMatrixLab/InferRSI/issues/3) · [Upstream PR](https://github.com/vllm-project/vllm-omni/pull/8739) - -### Ascend NPU: up to 22.0% reported latency reduction - -**Qwen-Image-2.1-Turbo · 1× Ascend 910B1 · 1280×704 · BF16 · Native 8 steps · eager** - -| Optimization | Mean vs 3111.5 ms baseline | Reported image validation | -| --- | :---: | --- | -| Conditioning reuse + interleaved RoPE | **2459.6 ms (−21.0%)** | Pixel-identical RGBA | -| + opt-in VAE fastpath | **2425.5 ms (−22.0%)** | Quality gate passed; not pixel-identical | - -Author-reported means from three requests after two warmups; not independently verified by this import. **The 30% target is not met.** The B300 and NPU cohorts use separate hardware and measurement protocols. - -[Ascend measurements and rejected rounds →](docs/qwen-image-2.1-turbo-ascend.md) · [Upstream NPU patch](https://github.com/vllm-project/vllm-omni/pull/8729) +## InferRSI x vLLM-Omni联合优化 + +InferRSI campaigns and upstream experiments cover image, video and speech models. +Each result applies to its linked workload and measured revisions. Some results +are reported directly by upstream contributors, with validation scoped to their benchmarks. + +| Model and hardware | Measured result and validation | Evidence | vLLM-Omni PR | +| --- | --- | --- | --- | +| **Qwen-Image-2.1-Turbo** · 4× B300 | Warm repeated-prompt HTTP end-to-end latency **384.16 → 266.60 ms (−30.60%)** in the historical InferRSI campaign. All 44 RGBA images were pixel-identical. | [Campaign #3](https://github.com/InferMatrixLab/InferRSI/issues/3) · [Pinned results](docs/qwen-image-2.1-turbo-b300.md) | [#8739](https://github.com/vllm-project/vllm-omni/pull/8739) (merged) | +| **Qwen-Image-2.1-Turbo** · 1× Ascend 910B1 | Upstream parent-to-merge comparison reported **22.20% lower warm repeated-prompt offline generation latency** and 18/18 pixel-identical RGBA images. Both arms aborted during teardown, leaving lifecycle `INCONCLUSIVE`. | [Earlier Ascend cohorts](docs/qwen-image-2.1-turbo-ascend.md) | [#8729](https://github.com/vllm-project/vllm-omni/pull/8729) (merged) | +| **MiniMax-H3 Ref2VA** · 8× Ascend 910B1 | For 5 s video, the tuned lossy Cache-DiT profile cut HTTP end-to-end latency **407.27 → 108.12 s (−73.45%)** versus dense inference. All 14 frozen quality comparisons passed. | [Campaign #293](https://github.com/InferMatrixLab/InferRSI/issues/293) · [Pinned results](evidence/h3-ref2va-ascend-293/results/final-5s/README.md) | [#9698](https://github.com/vllm-project/vllm-omni/pull/9698) (draft) | +| **MiniMax-H3 Ref2VA** · 4 logical Ascend 910C devices | One 5 s lossless screening pair with the migration and prefix-attention changes reported **291.25 → 286.67 s (−1.57%)** HTTP end-to-end. Strict quality checks passed for the screened candidate. Overall campaign `INCONCLUSIVE`. | [Campaign #297](https://github.com/InferMatrixLab/InferRSI/issues/297) · [Evidence PR #317](https://github.com/InferMatrixLab/InferRSI/pull/317) | [#9745](https://github.com/vllm-project/vllm-omni/pull/9745) (draft) | +| **MiniMax-H3-Image** · 4× Ascend 910B1 | With VAE graphs enabled versus the same-layout SP4 control, single-launch HTTP medians went **16.89 → 10.82 s** for 4736×3552 generation, **4.18 → 2.67 s** for 1024×1024 generation, and **5.68 → 4.57 s** for 1024×1024 editing. All 30 numerical image screens passed. Latency targets were unmet and final qualification remains `INCONCLUSIVE`. | [Campaign #390](https://github.com/InferMatrixLab/InferRSI/issues/390) · [Evidence PR #394](https://github.com/InferMatrixLab/InferRSI/pull/394) | [#10184](https://github.com/vllm-project/vllm-omni/pull/10184) (open) | +| **MiniMax-H3-Image** · 2× RTX 3090 | One upstream A/B of complete deployment profiles reported warm mean HTTP delivery **36.62 → 14.59 s (−60.16%)**. All nine paired images, including warmups, had identical decoded RGB pixels. Repeated qualification remains incomplete. | Benchmark and source pins in upstream PR | [#10243](https://github.com/vllm-project/vllm-omni/pull/10243) (open) | +| **MiniCPM-o 4.5** · 1× L20X | Packed conditioning added to a tuned configuration raised C32 throughput **15.55%** and reduced median time to first audio **27.69%** at the same 50-frame window in a historical screen. In a separate comparison, the optional TensorRT speaker + MPS profile raised throughput **20.5% at C16** and **25.4% at C32**, with speech quality verdict `FAIL`. | [Campaign #49](https://github.com/InferMatrixLab/InferRSI/issues/49) | [#9218](https://github.com/vllm-project/vllm-omni/pull/9218), [#9188](https://github.com/vllm-project/vllm-omni/pull/9188) (both merged) | +| **Qwen3-Omni** · 2× L20X | One MRv2 codec-autotuning pilot at C64 reported **7.79% higher throughput**, **6.06% lower mean real-time factor**, and **7.12% higher median time to first audio**. Campaign `INCONCLUSIVE`. | [Campaign #112](https://github.com/InferMatrixLab/InferRSI/issues/112) · [Pinned pilot](https://github.com/InferMatrixLab/InferRSI/tree/e1ae256fc075ea9ab64aec68da749cf4a8c22ac7/evidence/qwen3-omni-2gpu-20261023/results/pr9283-cudnn-performance) | [#9283](https://github.com/vllm-project/vllm-omni/pull/9283) (draft) | +| **LFM3-Omni-2B** · 1× Ascend 910B1 | Upstream offline Omni API A/B with depth graphs and grouped sampling reported **0.285 → 0.593 requests/s** at C4 and temperature 0. Retained audio waveforms matched bit-for-bit within each tested temperature. | Benchmark and source pins in upstream PR | [#9827](https://github.com/vllm-project/vllm-omni/pull/9827) (merged) | +| **LingBot-World** · 4× Ascend 910C1 | Upstream AB/BA benchmark reported full-request latency **108.29 → 103.89 s (−4.06%)** with whole-block KV gather. Pre-codec tensors and decoded RGB frames matched exactly. | Benchmark and source pins in upstream PR | [#8112](https://github.com/vllm-project/vllm-omni/pull/8112) (merged) | + +## Standalone engine benchmarks ### Standalone video/audio engines From 838fcb286fa5a2dc87f489578c146cb466cb575d Mon Sep 17 00:00:00 2001 From: zengchuang Date: Thu, 12 Nov 2026 15:08:46 +0800 Subject: [PATCH 02/11] docs: address review of vLLM-Omni optimization results Agent-assisted-by: Codex (author) Signed-off-by: zengchuang --- README.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 65baea1c2..3bd159076 100644 --- a/README.md +++ b/README.md @@ -63,7 +63,7 @@ For every CLI/Python operation — discovery, normalized reads, store candidates -## InferRSI x vLLM-Omni联合优化 +## InferRSI × vLLM-Omni optimization results InferRSI campaigns and upstream experiments cover image, video and speech models. Each result applies to its linked workload and measured revisions. Some results @@ -72,15 +72,17 @@ are reported directly by upstream contributors, with validation scoped to their | Model and hardware | Measured result and validation | Evidence | vLLM-Omni PR | | --- | --- | --- | --- | | **Qwen-Image-2.1-Turbo** · 4× B300 | Warm repeated-prompt HTTP end-to-end latency **384.16 → 266.60 ms (−30.60%)** in the historical InferRSI campaign. All 44 RGBA images were pixel-identical. | [Campaign #3](https://github.com/InferMatrixLab/InferRSI/issues/3) · [Pinned results](docs/qwen-image-2.1-turbo-b300.md) | [#8739](https://github.com/vllm-project/vllm-omni/pull/8739) (merged) | -| **Qwen-Image-2.1-Turbo** · 1× Ascend 910B1 | Upstream parent-to-merge comparison reported **22.20% lower warm repeated-prompt offline generation latency** and 18/18 pixel-identical RGBA images. Both arms aborted during teardown, leaving lifecycle `INCONCLUSIVE`. | [Earlier Ascend cohorts](docs/qwen-image-2.1-turbo-ascend.md) | [#8729](https://github.com/vllm-project/vllm-omni/pull/8729) (merged) | +| **Qwen-Image-2.1-Turbo** · 1× Ascend 910B1 | Upstream parent-to-merge comparison reported **22.20% lower warm repeated-prompt offline generation latency** and 18/18 pixel-identical RGBA images. The 30% reduction target was unmet. Both arms aborted during teardown, leaving lifecycle `INCONCLUSIVE`. | [Final-merge measurements](https://github.com/vllm-project/vllm-omni/pull/8729#:~:text=Final%2Dmerge%20performance%20and%20image%20comparison) · [Earlier Ascend cohorts](docs/qwen-image-2.1-turbo-ascend.md) | [#8729](https://github.com/vllm-project/vllm-omni/pull/8729) (merged) | | **MiniMax-H3 Ref2VA** · 8× Ascend 910B1 | For 5 s video, the tuned lossy Cache-DiT profile cut HTTP end-to-end latency **407.27 → 108.12 s (−73.45%)** versus dense inference. All 14 frozen quality comparisons passed. | [Campaign #293](https://github.com/InferMatrixLab/InferRSI/issues/293) · [Pinned results](evidence/h3-ref2va-ascend-293/results/final-5s/README.md) | [#9698](https://github.com/vllm-project/vllm-omni/pull/9698) (draft) | | **MiniMax-H3 Ref2VA** · 4 logical Ascend 910C devices | One 5 s lossless screening pair with the migration and prefix-attention changes reported **291.25 → 286.67 s (−1.57%)** HTTP end-to-end. Strict quality checks passed for the screened candidate. Overall campaign `INCONCLUSIVE`. | [Campaign #297](https://github.com/InferMatrixLab/InferRSI/issues/297) · [Evidence PR #317](https://github.com/InferMatrixLab/InferRSI/pull/317) | [#9745](https://github.com/vllm-project/vllm-omni/pull/9745) (draft) | | **MiniMax-H3-Image** · 4× Ascend 910B1 | With VAE graphs enabled versus the same-layout SP4 control, single-launch HTTP medians went **16.89 → 10.82 s** for 4736×3552 generation, **4.18 → 2.67 s** for 1024×1024 generation, and **5.68 → 4.57 s** for 1024×1024 editing. All 30 numerical image screens passed. Latency targets were unmet and final qualification remains `INCONCLUSIVE`. | [Campaign #390](https://github.com/InferMatrixLab/InferRSI/issues/390) · [Evidence PR #394](https://github.com/InferMatrixLab/InferRSI/pull/394) | [#10184](https://github.com/vllm-project/vllm-omni/pull/10184) (open) | -| **MiniMax-H3-Image** · 2× RTX 3090 | One upstream A/B of complete deployment profiles reported warm mean HTTP delivery **36.62 → 14.59 s (−60.16%)**. All nine paired images, including warmups, had identical decoded RGB pixels. Repeated qualification remains incomplete. | Benchmark and source pins in upstream PR | [#10243](https://github.com/vllm-project/vllm-omni/pull/10243) (open) | +| **MiniMax-H3-Image** · 2× RTX 3090 | One upstream A/B of complete deployment profiles reported warm mean HTTP delivery **36.62 → 14.59 s (−60.16%)**. All nine paired images, including warmups, had identical decoded RGB pixels. Repeated qualification remains incomplete. | [Paired deployment benchmark](https://github.com/vllm-project/vllm-omni/pull/10243#:~:text=One%20paired%20deployment%20comparison) | [#10243](https://github.com/vllm-project/vllm-omni/pull/10243) (open) | | **MiniCPM-o 4.5** · 1× L20X | Packed conditioning added to a tuned configuration raised C32 throughput **15.55%** and reduced median time to first audio **27.69%** at the same 50-frame window in a historical screen. In a separate comparison, the optional TensorRT speaker + MPS profile raised throughput **20.5% at C16** and **25.4% at C32**, with speech quality verdict `FAIL`. | [Campaign #49](https://github.com/InferMatrixLab/InferRSI/issues/49) | [#9218](https://github.com/vllm-project/vllm-omni/pull/9218), [#9188](https://github.com/vllm-project/vllm-omni/pull/9188) (both merged) | -| **Qwen3-Omni** · 2× L20X | One MRv2 codec-autotuning pilot at C64 reported **7.79% higher throughput**, **6.06% lower mean real-time factor**, and **7.12% higher median time to first audio**. Campaign `INCONCLUSIVE`. | [Campaign #112](https://github.com/InferMatrixLab/InferRSI/issues/112) · [Pinned pilot](https://github.com/InferMatrixLab/InferRSI/tree/e1ae256fc075ea9ab64aec68da749cf4a8c22ac7/evidence/qwen3-omni-2gpu-20261023/results/pr9283-cudnn-performance) | [#9283](https://github.com/vllm-project/vllm-omni/pull/9283) (draft) | -| **LFM3-Omni-2B** · 1× Ascend 910B1 | Upstream offline Omni API A/B with depth graphs and grouped sampling reported **0.285 → 0.593 requests/s** at C4 and temperature 0. Retained audio waveforms matched bit-for-bit within each tested temperature. | Benchmark and source pins in upstream PR | [#9827](https://github.com/vllm-project/vllm-omni/pull/9827) (merged) | -| **LingBot-World** · 4× Ascend 910C1 | Upstream AB/BA benchmark reported full-request latency **108.29 → 103.89 s (−4.06%)** with whole-block KV gather. Pre-codec tensors and decoded RGB frames matched exactly. | Benchmark and source pins in upstream PR | [#8112](https://github.com/vllm-project/vllm-omni/pull/8112) (merged) | +| **Qwen3-Omni** · 2× L20X | One MRv2 codec-autotuning pilot at C64 reported **7.79% higher throughput** and **6.06% lower mean real-time factor**. Median time to first audio **regressed by 7.12%**. Campaign `INCONCLUSIVE`. | [Campaign #112](https://github.com/InferMatrixLab/InferRSI/issues/112) · [Pinned pilot](https://github.com/InferMatrixLab/InferRSI/tree/e1ae256fc075ea9ab64aec68da749cf4a8c22ac7/evidence/qwen3-omni-2gpu-20261023/results/pr9283-cudnn-performance) | [#9283](https://github.com/vllm-project/vllm-omni/pull/9283) (draft) | +| **LFM3-Omni-2B** · 1× Ascend 910B1 | Upstream offline Omni API A/B with depth graphs and grouped sampling reported **0.285 → 0.593 requests/s** at C4 and temperature 0. Retained audio waveforms matched bit-for-bit within each tested temperature. | [Offline A/B benchmark](https://github.com/vllm-project/vllm-omni/pull/9827#:~:text=Performance%20results) | [#9827](https://github.com/vllm-project/vllm-omni/pull/9827) (merged) | +| **LingBot-World** · 4× Ascend 910C1 | Upstream AB/BA benchmark reported full-request latency **108.29 → 103.89 s (−4.06%)** with whole-block KV gather. Pre-codec tensors and decoded RGB frames matched exactly. | [KV gather benchmark](https://github.com/vllm-project/vllm-omni/pull/8112#:~:text=Performance%3A%20whole%2Dblock%20index_select%20KV%20gather%20on%20NPU) | [#8112](https://github.com/vllm-project/vllm-omni/pull/8112) (merged) | + +Upstream PR statuses checked on 2026-11-12. ## Standalone engine benchmarks From 236b76c0ef19743e1f85432a447bb0c4a9fa77cf Mon Sep 17 00:00:00 2001 From: zengchuang Date: Thu, 12 Nov 2026 15:36:07 +0800 Subject: [PATCH 03/11] docs: highlight standout vLLM-Omni performance gains Agent-assisted-by: Codex (author) Signed-off-by: zengchuang --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 3bd159076..38f2bf1bf 100644 --- a/README.md +++ b/README.md @@ -63,7 +63,7 @@ For every CLI/Python operation — discovery, normalized reads, store candidates -## InferRSI × vLLM-Omni optimization results +## InferRSI × vLLM-Omni: Standout Performance Gains InferRSI campaigns and upstream experiments cover image, video and speech models. Each result applies to its linked workload and measured revisions. Some results From b2e2b0a994f63569818b40fee926ad010720dfdf Mon Sep 17 00:00:00 2001 From: zengchuang Date: Thu, 12 Nov 2026 15:43:20 +0800 Subject: [PATCH 04/11] docs: separate optimization approaches and measured gains Agent-assisted-by: Codex (author) Signed-off-by: zengchuang --- README.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index 38f2bf1bf..8cd7632be 100644 --- a/README.md +++ b/README.md @@ -69,18 +69,18 @@ InferRSI campaigns and upstream experiments cover image, video and speech models Each result applies to its linked workload and measured revisions. Some results are reported directly by upstream contributors, with validation scoped to their benchmarks. -| Model and hardware | Measured result and validation | Evidence | vLLM-Omni PR | -| --- | --- | --- | --- | -| **Qwen-Image-2.1-Turbo** · 4× B300 | Warm repeated-prompt HTTP end-to-end latency **384.16 → 266.60 ms (−30.60%)** in the historical InferRSI campaign. All 44 RGBA images were pixel-identical. | [Campaign #3](https://github.com/InferMatrixLab/InferRSI/issues/3) · [Pinned results](docs/qwen-image-2.1-turbo-b300.md) | [#8739](https://github.com/vllm-project/vllm-omni/pull/8739) (merged) | -| **Qwen-Image-2.1-Turbo** · 1× Ascend 910B1 | Upstream parent-to-merge comparison reported **22.20% lower warm repeated-prompt offline generation latency** and 18/18 pixel-identical RGBA images. The 30% reduction target was unmet. Both arms aborted during teardown, leaving lifecycle `INCONCLUSIVE`. | [Final-merge measurements](https://github.com/vllm-project/vllm-omni/pull/8729#:~:text=Final%2Dmerge%20performance%20and%20image%20comparison) · [Earlier Ascend cohorts](docs/qwen-image-2.1-turbo-ascend.md) | [#8729](https://github.com/vllm-project/vllm-omni/pull/8729) (merged) | -| **MiniMax-H3 Ref2VA** · 8× Ascend 910B1 | For 5 s video, the tuned lossy Cache-DiT profile cut HTTP end-to-end latency **407.27 → 108.12 s (−73.45%)** versus dense inference. All 14 frozen quality comparisons passed. | [Campaign #293](https://github.com/InferMatrixLab/InferRSI/issues/293) · [Pinned results](evidence/h3-ref2va-ascend-293/results/final-5s/README.md) | [#9698](https://github.com/vllm-project/vllm-omni/pull/9698) (draft) | -| **MiniMax-H3 Ref2VA** · 4 logical Ascend 910C devices | One 5 s lossless screening pair with the migration and prefix-attention changes reported **291.25 → 286.67 s (−1.57%)** HTTP end-to-end. Strict quality checks passed for the screened candidate. Overall campaign `INCONCLUSIVE`. | [Campaign #297](https://github.com/InferMatrixLab/InferRSI/issues/297) · [Evidence PR #317](https://github.com/InferMatrixLab/InferRSI/pull/317) | [#9745](https://github.com/vllm-project/vllm-omni/pull/9745) (draft) | -| **MiniMax-H3-Image** · 4× Ascend 910B1 | With VAE graphs enabled versus the same-layout SP4 control, single-launch HTTP medians went **16.89 → 10.82 s** for 4736×3552 generation, **4.18 → 2.67 s** for 1024×1024 generation, and **5.68 → 4.57 s** for 1024×1024 editing. All 30 numerical image screens passed. Latency targets were unmet and final qualification remains `INCONCLUSIVE`. | [Campaign #390](https://github.com/InferMatrixLab/InferRSI/issues/390) · [Evidence PR #394](https://github.com/InferMatrixLab/InferRSI/pull/394) | [#10184](https://github.com/vllm-project/vllm-omni/pull/10184) (open) | -| **MiniMax-H3-Image** · 2× RTX 3090 | One upstream A/B of complete deployment profiles reported warm mean HTTP delivery **36.62 → 14.59 s (−60.16%)**. All nine paired images, including warmups, had identical decoded RGB pixels. Repeated qualification remains incomplete. | [Paired deployment benchmark](https://github.com/vllm-project/vllm-omni/pull/10243#:~:text=One%20paired%20deployment%20comparison) | [#10243](https://github.com/vllm-project/vllm-omni/pull/10243) (open) | -| **MiniCPM-o 4.5** · 1× L20X | Packed conditioning added to a tuned configuration raised C32 throughput **15.55%** and reduced median time to first audio **27.69%** at the same 50-frame window in a historical screen. In a separate comparison, the optional TensorRT speaker + MPS profile raised throughput **20.5% at C16** and **25.4% at C32**, with speech quality verdict `FAIL`. | [Campaign #49](https://github.com/InferMatrixLab/InferRSI/issues/49) | [#9218](https://github.com/vllm-project/vllm-omni/pull/9218), [#9188](https://github.com/vllm-project/vllm-omni/pull/9188) (both merged) | -| **Qwen3-Omni** · 2× L20X | One MRv2 codec-autotuning pilot at C64 reported **7.79% higher throughput** and **6.06% lower mean real-time factor**. Median time to first audio **regressed by 7.12%**. Campaign `INCONCLUSIVE`. | [Campaign #112](https://github.com/InferMatrixLab/InferRSI/issues/112) · [Pinned pilot](https://github.com/InferMatrixLab/InferRSI/tree/e1ae256fc075ea9ab64aec68da749cf4a8c22ac7/evidence/qwen3-omni-2gpu-20261023/results/pr9283-cudnn-performance) | [#9283](https://github.com/vllm-project/vllm-omni/pull/9283) (draft) | -| **LFM3-Omni-2B** · 1× Ascend 910B1 | Upstream offline Omni API A/B with depth graphs and grouped sampling reported **0.285 → 0.593 requests/s** at C4 and temperature 0. Retained audio waveforms matched bit-for-bit within each tested temperature. | [Offline A/B benchmark](https://github.com/vllm-project/vllm-omni/pull/9827#:~:text=Performance%20results) | [#9827](https://github.com/vllm-project/vllm-omni/pull/9827) (merged) | -| **LingBot-World** · 4× Ascend 910C1 | Upstream AB/BA benchmark reported full-request latency **108.29 → 103.89 s (−4.06%)** with whole-block KV gather. Pre-codec tensors and decoded RGB frames matched exactly. | [KV gather benchmark](https://github.com/vllm-project/vllm-omni/pull/8112#:~:text=Performance%3A%20whole%2Dblock%20index_select%20KV%20gather%20on%20NPU) | [#8112](https://github.com/vllm-project/vllm-omni/pull/8112) (merged) | +| Model | Hardware | Optimization approach | Measured gains | Validation and limits | Evidence | vLLM-Omni PR | +| --- | --- | --- | --- | --- | --- | --- | +| **Qwen-Image-2.1-Turbo** | 4× B300 | Reuse text conditioning, fuse BF16 operations, and use symmetric-memory Ulysses with batched QKV exchange. | Warm repeated-prompt HTTP end-to-end latency **384.16 → 266.60 ms (−30.60%)** in the historical InferRSI campaign. | All 44 RGBA images were pixel-identical. | [Campaign #3](https://github.com/InferMatrixLab/InferRSI/issues/3) · [Pinned results](docs/qwen-image-2.1-turbo-b300.md) | [#8739](https://github.com/vllm-project/vllm-omni/pull/8739) (merged) | +| **Qwen-Image-2.1-Turbo** | 1× Ascend 910B1 | Cache conditioning for repeated prompts, prepare shared modulation once per denoising step, and reuse frequency tables in native interleaved NPU RoPE. | **22.20% lower warm repeated-prompt offline generation latency** in the upstream parent-to-merge comparison. | 18/18 RGBA images were pixel-identical. The 30% target was unmet. Both arms aborted during teardown, leaving lifecycle `INCONCLUSIVE`. | [Final-merge measurements](https://github.com/vllm-project/vllm-omni/pull/8729#:~:text=Final%2Dmerge%20performance%20and%20image%20comparison) · [Earlier Ascend cohorts](docs/qwen-image-2.1-turbo-ascend.md) | [#8729](https://github.com/vllm-project/vllm-omni/pull/8729) (merged) | +| **MiniMax-H3 Ref2VA** | 8× Ascend 910B1 | Tune the lossy Cache-DiT profile. | For 5 s video, HTTP end-to-end latency **407.27 → 108.12 s (−73.45%)** versus dense inference. | All 14 frozen quality comparisons passed. | [Campaign #293](https://github.com/InferMatrixLab/InferRSI/issues/293) · [Pinned results](evidence/h3-ref2va-ascend-293/results/final-5s/README.md) | [#9698](https://github.com/vllm-project/vllm-omni/pull/9698) (draft) | +| **MiniMax-H3 Ref2VA** | 4 logical Ascend 910C devices | Combine the migration changes with opt-in BF16 prefix attention. | One 5 s lossless screening pair reported HTTP end-to-end latency **291.25 → 286.67 s (−1.57%)**. | Strict quality checks passed for the screened candidate. Overall campaign `INCONCLUSIVE`. | [Campaign #297](https://github.com/InferMatrixLab/InferRSI/issues/297) · [Evidence PR #317](https://github.com/InferMatrixLab/InferRSI/pull/317) | [#9745](https://github.com/vllm-project/vllm-omni/pull/9745) (draft) | +| **MiniMax-H3-Image** | 4× Ascend 910B1 | Enable opt-in VAE tile graphs and approximate encoder/decoder tile batching. | Single-launch HTTP medians versus the same-layout SP4 control were **16.89 → 10.82 s** for 4736×3552 generation, **4.18 → 2.67 s** for 1024×1024 generation, and **5.68 → 4.57 s** for 1024×1024 editing. | All 30 numerical image screens passed. Latency targets were unmet and final qualification remains `INCONCLUSIVE`. | [Campaign #390](https://github.com/InferMatrixLab/InferRSI/issues/390) · [Evidence PR #394](https://github.com/InferMatrixLab/InferRSI/pull/394) | [#10184](https://github.com/vllm-project/vllm-omni/pull/10184) (open) | +| **MiniMax-H3-Image** | 2× RTX 3090 | Combine rank-local layerwise offload with selected DiT weights kept resident, AdaLN caching, and reuse of request-local text refinement. | One upstream A/B of complete deployment profiles reported warm mean HTTP delivery **36.62 → 14.59 s (−60.16%)**. | All nine paired images, including warmups, had identical decoded RGB pixels. Repeated qualification remains incomplete. | [Paired deployment benchmark](https://github.com/vllm-project/vllm-omni/pull/10243#:~:text=One%20paired%20deployment%20comparison) | [#10243](https://github.com/vllm-project/vllm-omni/pull/10243) (open) | +| **MiniCPM-o 4.5** | 1× L20X | Add packed conditioning to a tuned configuration. A separate comparison enables TensorRT speaker extraction with CUDA MPS. | Packed conditioning at C32 and the same 50-frame window gave **15.55% higher throughput** and **27.69% lower median time to first audio**. TensorRT + MPS gave **20.5% higher throughput at C16** and **25.4% at C32**. | Packed conditioning was a historical screen. The TensorRT + MPS comparison had speech quality verdict `FAIL`. | [Campaign #49](https://github.com/InferMatrixLab/InferRSI/issues/49) | [#9218](https://github.com/vllm-project/vllm-omni/pull/9218), [#9188](https://github.com/vllm-project/vllm-omni/pull/9188) (both merged) | +| **Qwen3-Omni** | 2× L20X | Enable cuDNN codec autotuning in the MRv2 deployment. | One C64 pilot reported **7.79% higher throughput** and **6.06% lower mean real-time factor**. | Median time to first audio **regressed by 7.12%**. Campaign `INCONCLUSIVE`. | [Campaign #112](https://github.com/InferMatrixLab/InferRSI/issues/112) · [Pinned pilot](https://github.com/InferMatrixLab/InferRSI/tree/e1ae256fc075ea9ab64aec68da749cf4a8c22ac7/evidence/qwen3-omni-2gpu-20261023/results/pr9283-cudnn-performance) | [#9283](https://github.com/vllm-project/vllm-omni/pull/9283) (draft) | +| **LFM3-Omni-2B** | 1× Ascend 910B1 | Enable depth graphs and grouped greedy sampling. | Upstream offline Omni API A/B reported **0.285 → 0.593 requests/s** at C4 and temperature 0. | Retained audio waveforms matched bit-for-bit within each tested temperature. | [Offline A/B benchmark](https://github.com/vllm-project/vllm-omni/pull/9827#:~:text=Performance%20results) | [#9827](https://github.com/vllm-project/vllm-omni/pull/9827) (merged) | +| **LingBot-World** | 4× Ascend 910C1 | Gather paged KV in whole blocks with `index_select`. | Upstream AB/BA reported full-request latency **108.29 → 103.89 s (−4.06%)**. | Pre-codec tensors and decoded RGB frames matched exactly. | [KV gather benchmark](https://github.com/vllm-project/vllm-omni/pull/8112#:~:text=Performance%3A%20whole%2Dblock%20index_select%20KV%20gather%20on%20NPU) | [#8112](https://github.com/vllm-project/vllm-omni/pull/8112) (merged) | Upstream PR statuses checked on 2026-11-12. From 72b3ca5d9296cebcf1216a803b42f32ade857791 Mon Sep 17 00:00:00 2001 From: zengchuang Date: Thu, 12 Nov 2026 15:52:17 +0800 Subject: [PATCH 05/11] docs: balance performance highlights with trade-offs Agent-assisted-by: Codex (author) Signed-off-by: zengchuang --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 8cd7632be..4a84f5866 100644 --- a/README.md +++ b/README.md @@ -63,7 +63,7 @@ For every CLI/Python operation — discovery, normalized reads, store candidates -## InferRSI × vLLM-Omni: Standout Performance Gains +## InferRSI × vLLM-Omni: Performance Highlights and Trade-offs InferRSI campaigns and upstream experiments cover image, video and speech models. Each result applies to its linked workload and measured revisions. Some results From 26da90c9cb3a6f0e5f4e16fe70b123112eadf494 Mon Sep 17 00:00:00 2001 From: zengchuang Date: Thu, 12 Nov 2026 18:04:26 +0800 Subject: [PATCH 06/11] docs: emphasize measured gains and list optimization approaches Agent-assisted-by: Codex (author) Signed-off-by: zengchuang --- README.md | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 4a84f5866..dc3e19443 100644 --- a/README.md +++ b/README.md @@ -71,16 +71,18 @@ are reported directly by upstream contributors, with validation scoped to their | Model | Hardware | Optimization approach | Measured gains | Validation and limits | Evidence | vLLM-Omni PR | | --- | --- | --- | --- | --- | --- | --- | -| **Qwen-Image-2.1-Turbo** | 4× B300 | Reuse text conditioning, fuse BF16 operations, and use symmetric-memory Ulysses with batched QKV exchange. | Warm repeated-prompt HTTP end-to-end latency **384.16 → 266.60 ms (−30.60%)** in the historical InferRSI campaign. | All 44 RGBA images were pixel-identical. | [Campaign #3](https://github.com/InferMatrixLab/InferRSI/issues/3) · [Pinned results](docs/qwen-image-2.1-turbo-b300.md) | [#8739](https://github.com/vllm-project/vllm-omni/pull/8739) (merged) | -| **Qwen-Image-2.1-Turbo** | 1× Ascend 910B1 | Cache conditioning for repeated prompts, prepare shared modulation once per denoising step, and reuse frequency tables in native interleaved NPU RoPE. | **22.20% lower warm repeated-prompt offline generation latency** in the upstream parent-to-merge comparison. | 18/18 RGBA images were pixel-identical. The 30% target was unmet. Both arms aborted during teardown, leaving lifecycle `INCONCLUSIVE`. | [Final-merge measurements](https://github.com/vllm-project/vllm-omni/pull/8729#:~:text=Final%2Dmerge%20performance%20and%20image%20comparison) · [Earlier Ascend cohorts](docs/qwen-image-2.1-turbo-ascend.md) | [#8729](https://github.com/vllm-project/vllm-omni/pull/8729) (merged) | -| **MiniMax-H3 Ref2VA** | 8× Ascend 910B1 | Tune the lossy Cache-DiT profile. | For 5 s video, HTTP end-to-end latency **407.27 → 108.12 s (−73.45%)** versus dense inference. | All 14 frozen quality comparisons passed. | [Campaign #293](https://github.com/InferMatrixLab/InferRSI/issues/293) · [Pinned results](evidence/h3-ref2va-ascend-293/results/final-5s/README.md) | [#9698](https://github.com/vllm-project/vllm-omni/pull/9698) (draft) | -| **MiniMax-H3 Ref2VA** | 4 logical Ascend 910C devices | Combine the migration changes with opt-in BF16 prefix attention. | One 5 s lossless screening pair reported HTTP end-to-end latency **291.25 → 286.67 s (−1.57%)**. | Strict quality checks passed for the screened candidate. Overall campaign `INCONCLUSIVE`. | [Campaign #297](https://github.com/InferMatrixLab/InferRSI/issues/297) · [Evidence PR #317](https://github.com/InferMatrixLab/InferRSI/pull/317) | [#9745](https://github.com/vllm-project/vllm-omni/pull/9745) (draft) | -| **MiniMax-H3-Image** | 4× Ascend 910B1 | Enable opt-in VAE tile graphs and approximate encoder/decoder tile batching. | Single-launch HTTP medians versus the same-layout SP4 control were **16.89 → 10.82 s** for 4736×3552 generation, **4.18 → 2.67 s** for 1024×1024 generation, and **5.68 → 4.57 s** for 1024×1024 editing. | All 30 numerical image screens passed. Latency targets were unmet and final qualification remains `INCONCLUSIVE`. | [Campaign #390](https://github.com/InferMatrixLab/InferRSI/issues/390) · [Evidence PR #394](https://github.com/InferMatrixLab/InferRSI/pull/394) | [#10184](https://github.com/vllm-project/vllm-omni/pull/10184) (open) | -| **MiniMax-H3-Image** | 2× RTX 3090 | Combine rank-local layerwise offload with selected DiT weights kept resident, AdaLN caching, and reuse of request-local text refinement. | One upstream A/B of complete deployment profiles reported warm mean HTTP delivery **36.62 → 14.59 s (−60.16%)**. | All nine paired images, including warmups, had identical decoded RGB pixels. Repeated qualification remains incomplete. | [Paired deployment benchmark](https://github.com/vllm-project/vllm-omni/pull/10243#:~:text=One%20paired%20deployment%20comparison) | [#10243](https://github.com/vllm-project/vllm-omni/pull/10243) (open) | -| **MiniCPM-o 4.5** | 1× L20X | Add packed conditioning to a tuned configuration. A separate comparison enables TensorRT speaker extraction with CUDA MPS. | Packed conditioning at C32 and the same 50-frame window gave **15.55% higher throughput** and **27.69% lower median time to first audio**. TensorRT + MPS gave **20.5% higher throughput at C16** and **25.4% at C32**. | Packed conditioning was a historical screen. The TensorRT + MPS comparison had speech quality verdict `FAIL`. | [Campaign #49](https://github.com/InferMatrixLab/InferRSI/issues/49) | [#9218](https://github.com/vllm-project/vllm-omni/pull/9218), [#9188](https://github.com/vllm-project/vllm-omni/pull/9188) (both merged) | -| **Qwen3-Omni** | 2× L20X | Enable cuDNN codec autotuning in the MRv2 deployment. | One C64 pilot reported **7.79% higher throughput** and **6.06% lower mean real-time factor**. | Median time to first audio **regressed by 7.12%**. Campaign `INCONCLUSIVE`. | [Campaign #112](https://github.com/InferMatrixLab/InferRSI/issues/112) · [Pinned pilot](https://github.com/InferMatrixLab/InferRSI/tree/e1ae256fc075ea9ab64aec68da749cf4a8c22ac7/evidence/qwen3-omni-2gpu-20261023/results/pr9283-cudnn-performance) | [#9283](https://github.com/vllm-project/vllm-omni/pull/9283) (draft) | -| **LFM3-Omni-2B** | 1× Ascend 910B1 | Enable depth graphs and grouped greedy sampling. | Upstream offline Omni API A/B reported **0.285 → 0.593 requests/s** at C4 and temperature 0. | Retained audio waveforms matched bit-for-bit within each tested temperature. | [Offline A/B benchmark](https://github.com/vllm-project/vllm-omni/pull/9827#:~:text=Performance%20results) | [#9827](https://github.com/vllm-project/vllm-omni/pull/9827) (merged) | -| **LingBot-World** | 4× Ascend 910C1 | Gather paged KV in whole blocks with `index_select`. | Upstream AB/BA reported full-request latency **108.29 → 103.89 s (−4.06%)**. | Pre-codec tensors and decoded RGB frames matched exactly. | [KV gather benchmark](https://github.com/vllm-project/vllm-omni/pull/8112#:~:text=Performance%3A%20whole%2Dblock%20index_select%20KV%20gather%20on%20NPU) | [#8112](https://github.com/vllm-project/vllm-omni/pull/8112) (merged) | +| **Qwen-Image-2.1-Turbo** | 4× B300 |
  • Text-conditioning reuse
  • BF16 operator fusion
  • Symmetric-memory Ulysses with batched QKV exchange
| **384.16 → 266.60 ms (−30.60%)**
Warm HTTP end-to-end latency, repeated prompts | Historical InferRSI campaign. All 44 RGBA images were pixel-identical. | [Campaign #3](https://github.com/InferMatrixLab/InferRSI/issues/3) · [Pinned results](docs/qwen-image-2.1-turbo-b300.md) | [#8739](https://github.com/vllm-project/vllm-omni/pull/8739) (merged) | +| **Qwen-Image-2.1-Turbo** | 1× Ascend 910B1 |
  • Conditioning cache for repeated prompts
  • Shared modulation once per denoising step
  • Native interleaved NPU RoPE with frequency-table reuse
| **−22.20%** offline generation latency
Warm repeated prompts | Upstream parent-to-merge comparison. 18/18 RGBA images were pixel-identical. The 30% target was unmet. Both arms aborted during teardown, leaving lifecycle `INCONCLUSIVE`. | [Final-merge measurements](https://github.com/vllm-project/vllm-omni/pull/8729#:~:text=Final%2Dmerge%20performance%20and%20image%20comparison) · [Earlier Ascend cohorts](docs/qwen-image-2.1-turbo-ascend.md) | [#8729](https://github.com/vllm-project/vllm-omni/pull/8729) (merged) | +| **MiniMax-H3 Ref2VA** | 8× Ascend 910B1 |
  • Lossy Cache-DiT profile tuning
| **407.27 → 108.12 s (−73.45%)**
5 s video, HTTP end-to-end vs. dense inference | All 14 frozen quality comparisons passed. | [Campaign #293](https://github.com/InferMatrixLab/InferRSI/issues/293) · [Pinned results](evidence/h3-ref2va-ascend-293/results/final-5s/README.md) | [#9698](https://github.com/vllm-project/vllm-omni/pull/9698) (draft) | +| **MiniMax-H3 Ref2VA** | 4 logical Ascend 910C devices |
  • Ref2VA migration changes
  • Opt-in BF16 prefix attention
| **291.25 → 286.67 s (−1.57%)**
5 s video, HTTP end-to-end | One lossless screening pair. Strict quality checks passed for the screened candidate. Overall campaign `INCONCLUSIVE`. | [Campaign #297](https://github.com/InferMatrixLab/InferRSI/issues/297) · [Evidence PR #317](https://github.com/InferMatrixLab/InferRSI/pull/317) | [#9745](https://github.com/vllm-project/vllm-omni/pull/9745) (draft) | +| **MiniMax-H3-Image** | 4× Ascend 910B1 |
  • Opt-in VAE tile graphs
  • Approximate encoder/decoder tile batching
| **16.89 → 10.82 s** · 4736×3552 generation
**4.18 → 2.67 s** · 1024×1024 generation
**5.68 → 4.57 s** · 1024×1024 editing | Single-launch HTTP medians vs. the same-layout SP4 control. All 30 numerical image screens passed. Latency targets were unmet and final qualification remains `INCONCLUSIVE`. | [Campaign #390](https://github.com/InferMatrixLab/InferRSI/issues/390) · [Evidence PR #394](https://github.com/InferMatrixLab/InferRSI/pull/394) | [#10184](https://github.com/vllm-project/vllm-omni/pull/10184) (open) | +| **MiniMax-H3-Image** | 2× RTX 3090 |
  • Rank-local layerwise offload
  • Selected DiT weights kept resident
  • AdaLN caching
  • Request-local text refinement reuse
| **36.62 → 14.59 s (−60.16%)**
Warm mean HTTP delivery latency | One upstream A/B of complete deployment profiles. All nine paired RGB images, including warmups, matched exactly. Repeated qualification remains incomplete. | [Paired deployment benchmark](https://github.com/vllm-project/vllm-omni/pull/10243#:~:text=One%20paired%20deployment%20comparison) | [#10243](https://github.com/vllm-project/vllm-omni/pull/10243) (open) | +| **MiniCPM-o 4.5** | 1× L20X |
  • Packed conditioning on a tuned configuration
  • TensorRT speaker extraction + CUDA MPS (separate comparison)
| **+15.55%** throughput · **−27.69%** median TTFA
Packed conditioning, C32, same 50-frame window
**+20.5%** at C16 · **+25.4%** at C32
TensorRT + MPS throughput | Packed conditioning was a historical screen. The TensorRT + MPS comparison had speech quality verdict `FAIL`. | [Campaign #49](https://github.com/InferMatrixLab/InferRSI/issues/49) | [#9218](https://github.com/vllm-project/vllm-omni/pull/9218), [#9188](https://github.com/vllm-project/vllm-omni/pull/9188) (both merged) | +| **Qwen3-Omni** | 2× L20X |
  • cuDNN codec autotuning in the MRv2 deployment
| **+7.79%** throughput · **−6.06%** mean RTF
One C64 pilot | Median time to first audio **regressed by 7.12%**. Campaign `INCONCLUSIVE`. | [Campaign #112](https://github.com/InferMatrixLab/InferRSI/issues/112) · [Pinned pilot](https://github.com/InferMatrixLab/InferRSI/tree/e1ae256fc075ea9ab64aec68da749cf4a8c22ac7/evidence/qwen3-omni-2gpu-20261023/results/pr9283-cudnn-performance) | [#9283](https://github.com/vllm-project/vllm-omni/pull/9283) (draft) | +| **LFM3-Omni-2B** | 1× Ascend 910B1 |
  • Depth graphs
  • Grouped greedy sampling
| **0.285 → 0.593 requests/s**
Offline Omni API A/B, C4, temperature 0 | Retained audio waveforms matched bit-for-bit within each tested temperature. | [Offline A/B benchmark](https://github.com/vllm-project/vllm-omni/pull/9827#:~:text=Performance%20results) | [#9827](https://github.com/vllm-project/vllm-omni/pull/9827) (merged) | +| **LingBot-World** | 4× Ascend 910C1 |
  • Whole-block paged KV gather with index_select
| **108.29 → 103.89 s (−4.06%)**
Full-request latency, AB/BA | Pre-codec tensors and decoded RGB frames matched exactly. | [KV gather benchmark](https://github.com/vllm-project/vllm-omni/pull/8112#:~:text=Performance%3A%20whole%2Dblock%20index_select%20KV%20gather%20on%20NPU) | [#8112](https://github.com/vllm-project/vllm-omni/pull/8112) (merged) | + +TTFA is time to first audio. RTF is real-time factor. Upstream PR statuses checked on 2026-11-12. From 4cee566c0caf6c407d32053864e893ddba3bff33 Mon Sep 17 00:00:00 2001 From: zengchuang Date: Thu, 12 Nov 2026 18:04:58 +0800 Subject: [PATCH 07/11] docs: retain benchmark attribution and pixel validation scope Agent-assisted-by: Codex (author) Signed-off-by: zengchuang --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index dc3e19443..091ddbd9d 100644 --- a/README.md +++ b/README.md @@ -76,11 +76,11 @@ are reported directly by upstream contributors, with validation scoped to their | **MiniMax-H3 Ref2VA** | 8× Ascend 910B1 |
  • Lossy Cache-DiT profile tuning
| **407.27 → 108.12 s (−73.45%)**
5 s video, HTTP end-to-end vs. dense inference | All 14 frozen quality comparisons passed. | [Campaign #293](https://github.com/InferMatrixLab/InferRSI/issues/293) · [Pinned results](evidence/h3-ref2va-ascend-293/results/final-5s/README.md) | [#9698](https://github.com/vllm-project/vllm-omni/pull/9698) (draft) | | **MiniMax-H3 Ref2VA** | 4 logical Ascend 910C devices |
  • Ref2VA migration changes
  • Opt-in BF16 prefix attention
| **291.25 → 286.67 s (−1.57%)**
5 s video, HTTP end-to-end | One lossless screening pair. Strict quality checks passed for the screened candidate. Overall campaign `INCONCLUSIVE`. | [Campaign #297](https://github.com/InferMatrixLab/InferRSI/issues/297) · [Evidence PR #317](https://github.com/InferMatrixLab/InferRSI/pull/317) | [#9745](https://github.com/vllm-project/vllm-omni/pull/9745) (draft) | | **MiniMax-H3-Image** | 4× Ascend 910B1 |
  • Opt-in VAE tile graphs
  • Approximate encoder/decoder tile batching
| **16.89 → 10.82 s** · 4736×3552 generation
**4.18 → 2.67 s** · 1024×1024 generation
**5.68 → 4.57 s** · 1024×1024 editing | Single-launch HTTP medians vs. the same-layout SP4 control. All 30 numerical image screens passed. Latency targets were unmet and final qualification remains `INCONCLUSIVE`. | [Campaign #390](https://github.com/InferMatrixLab/InferRSI/issues/390) · [Evidence PR #394](https://github.com/InferMatrixLab/InferRSI/pull/394) | [#10184](https://github.com/vllm-project/vllm-omni/pull/10184) (open) | -| **MiniMax-H3-Image** | 2× RTX 3090 |
  • Rank-local layerwise offload
  • Selected DiT weights kept resident
  • AdaLN caching
  • Request-local text refinement reuse
| **36.62 → 14.59 s (−60.16%)**
Warm mean HTTP delivery latency | One upstream A/B of complete deployment profiles. All nine paired RGB images, including warmups, matched exactly. Repeated qualification remains incomplete. | [Paired deployment benchmark](https://github.com/vllm-project/vllm-omni/pull/10243#:~:text=One%20paired%20deployment%20comparison) | [#10243](https://github.com/vllm-project/vllm-omni/pull/10243) (open) | +| **MiniMax-H3-Image** | 2× RTX 3090 |
  • Rank-local layerwise offload
  • Selected DiT weights kept resident
  • AdaLN caching
  • Request-local text refinement reuse
| **36.62 → 14.59 s (−60.16%)**
Warm mean HTTP delivery latency | One upstream A/B of complete deployment profiles. All nine image pairs had identical decoded RGB pixels, including warmups. Repeated qualification remains incomplete. | [Paired deployment benchmark](https://github.com/vllm-project/vllm-omni/pull/10243#:~:text=One%20paired%20deployment%20comparison) | [#10243](https://github.com/vllm-project/vllm-omni/pull/10243) (open) | | **MiniCPM-o 4.5** | 1× L20X |
  • Packed conditioning on a tuned configuration
  • TensorRT speaker extraction + CUDA MPS (separate comparison)
| **+15.55%** throughput · **−27.69%** median TTFA
Packed conditioning, C32, same 50-frame window
**+20.5%** at C16 · **+25.4%** at C32
TensorRT + MPS throughput | Packed conditioning was a historical screen. The TensorRT + MPS comparison had speech quality verdict `FAIL`. | [Campaign #49](https://github.com/InferMatrixLab/InferRSI/issues/49) | [#9218](https://github.com/vllm-project/vllm-omni/pull/9218), [#9188](https://github.com/vllm-project/vllm-omni/pull/9188) (both merged) | | **Qwen3-Omni** | 2× L20X |
  • cuDNN codec autotuning in the MRv2 deployment
| **+7.79%** throughput · **−6.06%** mean RTF
One C64 pilot | Median time to first audio **regressed by 7.12%**. Campaign `INCONCLUSIVE`. | [Campaign #112](https://github.com/InferMatrixLab/InferRSI/issues/112) · [Pinned pilot](https://github.com/InferMatrixLab/InferRSI/tree/e1ae256fc075ea9ab64aec68da749cf4a8c22ac7/evidence/qwen3-omni-2gpu-20261023/results/pr9283-cudnn-performance) | [#9283](https://github.com/vllm-project/vllm-omni/pull/9283) (draft) | -| **LFM3-Omni-2B** | 1× Ascend 910B1 |
  • Depth graphs
  • Grouped greedy sampling
| **0.285 → 0.593 requests/s**
Offline Omni API A/B, C4, temperature 0 | Retained audio waveforms matched bit-for-bit within each tested temperature. | [Offline A/B benchmark](https://github.com/vllm-project/vllm-omni/pull/9827#:~:text=Performance%20results) | [#9827](https://github.com/vllm-project/vllm-omni/pull/9827) (merged) | -| **LingBot-World** | 4× Ascend 910C1 |
  • Whole-block paged KV gather with index_select
| **108.29 → 103.89 s (−4.06%)**
Full-request latency, AB/BA | Pre-codec tensors and decoded RGB frames matched exactly. | [KV gather benchmark](https://github.com/vllm-project/vllm-omni/pull/8112#:~:text=Performance%3A%20whole%2Dblock%20index_select%20KV%20gather%20on%20NPU) | [#8112](https://github.com/vllm-project/vllm-omni/pull/8112) (merged) | +| **LFM3-Omni-2B** | 1× Ascend 910B1 |
  • Depth graphs
  • Grouped greedy sampling
| **0.285 → 0.593 requests/s**
Upstream offline Omni API A/B, C4, temperature 0 | Retained audio waveforms matched bit-for-bit within each tested temperature. | [Offline A/B benchmark](https://github.com/vllm-project/vllm-omni/pull/9827#:~:text=Performance%20results) | [#9827](https://github.com/vllm-project/vllm-omni/pull/9827) (merged) | +| **LingBot-World** | 4× Ascend 910C1 |
  • Whole-block paged KV gather with index_select
| **108.29 → 103.89 s (−4.06%)**
Full-request latency, upstream AB/BA | Pre-codec tensors and decoded RGB frames matched exactly. | [KV gather benchmark](https://github.com/vllm-project/vllm-omni/pull/8112#:~:text=Performance%3A%20whole%2Dblock%20index_select%20KV%20gather%20on%20NPU) | [#8112](https://github.com/vllm-project/vllm-omni/pull/8112) (merged) | TTFA is time to first audio. RTF is real-time factor. From 52da29412dfbbee67d682c8f1bdf14b16139ca52 Mon Sep 17 00:00:00 2001 From: zengchuang Date: Thu, 12 Nov 2026 19:32:55 +0800 Subject: [PATCH 08/11] docs: make the optimization benchmark table collapsible Agent-assisted-by: Codex (author) Signed-off-by: zengchuang --- README.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/README.md b/README.md index 091ddbd9d..4cee62a45 100644 --- a/README.md +++ b/README.md @@ -69,6 +69,9 @@ InferRSI campaigns and upstream experiments cover image, video and speech models Each result applies to its linked workload and measured revisions. Some results are reported directly by upstream contributors, with validation scoped to their benchmarks. +
+View model benchmarks and upstream PRs + | Model | Hardware | Optimization approach | Measured gains | Validation and limits | Evidence | vLLM-Omni PR | | --- | --- | --- | --- | --- | --- | --- | | **Qwen-Image-2.1-Turbo** | 4× B300 |
  • Text-conditioning reuse
  • BF16 operator fusion
  • Symmetric-memory Ulysses with batched QKV exchange
| **384.16 → 266.60 ms (−30.60%)**
Warm HTTP end-to-end latency, repeated prompts | Historical InferRSI campaign. All 44 RGBA images were pixel-identical. | [Campaign #3](https://github.com/InferMatrixLab/InferRSI/issues/3) · [Pinned results](docs/qwen-image-2.1-turbo-b300.md) | [#8739](https://github.com/vllm-project/vllm-omni/pull/8739) (merged) | @@ -86,6 +89,8 @@ TTFA is time to first audio. RTF is real-time factor. Upstream PR statuses checked on 2026-11-12. +
+ ## Standalone engine benchmarks ### Standalone video/audio engines From 290b1e6db058184acd460bba621a7b2040796ffd Mon Sep 17 00:00:00 2001 From: zengchuang Date: Thu, 12 Nov 2026 19:36:29 +0800 Subject: [PATCH 09/11] docs: align README section icons and shorten the performance heading Agent-assisted-by: Codex (author) Signed-off-by: zengchuang --- README.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 4cee62a45..2019e3a4e 100644 --- a/README.md +++ b/README.md @@ -51,7 +51,9 @@ The **offline harness-improvement MVP** adds a separate Skill Eval path for supp declarative candidates. Its fixed-output experiments validate the controller; real-model gains remain unmeasured. [Flow, limits and reproduction →](docs/harness-improvement.md) -## Choose your starting path + + +## 🧭 Choose your starting path | You want to… | Start at | | --- | --- | @@ -63,7 +65,7 @@ For every CLI/Python operation — discovery, normalized reads, store candidates -## InferRSI × vLLM-Omni: Performance Highlights and Trade-offs +## 📊 InferRSI × vLLM-Omni: Performance Highlights InferRSI campaigns and upstream experiments cover image, video and speech models. Each result applies to its linked workload and measured revisions. Some results @@ -91,7 +93,9 @@ Upstream PR statuses checked on 2026-11-12. -## Standalone engine benchmarks + + +## 🧪 Standalone engine benchmarks ### Standalone video/audio engines From 72bdd6c3ea5c0dda35cc05f1b72a89ba296ed454 Mon Sep 17 00:00:00 2001 From: zengchuang Date: Thu, 12 Nov 2026 19:43:39 +0800 Subject: [PATCH 10/11] docs: narrow performance highlights and expand the table by default Agent-assisted-by: Codex (author) Signed-off-by: zengchuang --- README.md | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 2019e3a4e..57f495fce 100644 --- a/README.md +++ b/README.md @@ -71,24 +71,17 @@ InferRSI campaigns and upstream experiments cover image, video and speech models Each result applies to its linked workload and measured revisions. Some results are reported directly by upstream contributors, with validation scoped to their benchmarks. -
-View model benchmarks and upstream PRs +
+Benchmarks, validation limits, and upstream PRs | Model | Hardware | Optimization approach | Measured gains | Validation and limits | Evidence | vLLM-Omni PR | | --- | --- | --- | --- | --- | --- | --- | | **Qwen-Image-2.1-Turbo** | 4× B300 |
  • Text-conditioning reuse
  • BF16 operator fusion
  • Symmetric-memory Ulysses with batched QKV exchange
| **384.16 → 266.60 ms (−30.60%)**
Warm HTTP end-to-end latency, repeated prompts | Historical InferRSI campaign. All 44 RGBA images were pixel-identical. | [Campaign #3](https://github.com/InferMatrixLab/InferRSI/issues/3) · [Pinned results](docs/qwen-image-2.1-turbo-b300.md) | [#8739](https://github.com/vllm-project/vllm-omni/pull/8739) (merged) | -| **Qwen-Image-2.1-Turbo** | 1× Ascend 910B1 |
  • Conditioning cache for repeated prompts
  • Shared modulation once per denoising step
  • Native interleaved NPU RoPE with frequency-table reuse
| **−22.20%** offline generation latency
Warm repeated prompts | Upstream parent-to-merge comparison. 18/18 RGBA images were pixel-identical. The 30% target was unmet. Both arms aborted during teardown, leaving lifecycle `INCONCLUSIVE`. | [Final-merge measurements](https://github.com/vllm-project/vllm-omni/pull/8729#:~:text=Final%2Dmerge%20performance%20and%20image%20comparison) · [Earlier Ascend cohorts](docs/qwen-image-2.1-turbo-ascend.md) | [#8729](https://github.com/vllm-project/vllm-omni/pull/8729) (merged) | | **MiniMax-H3 Ref2VA** | 8× Ascend 910B1 |
  • Lossy Cache-DiT profile tuning
| **407.27 → 108.12 s (−73.45%)**
5 s video, HTTP end-to-end vs. dense inference | All 14 frozen quality comparisons passed. | [Campaign #293](https://github.com/InferMatrixLab/InferRSI/issues/293) · [Pinned results](evidence/h3-ref2va-ascend-293/results/final-5s/README.md) | [#9698](https://github.com/vllm-project/vllm-omni/pull/9698) (draft) | -| **MiniMax-H3 Ref2VA** | 4 logical Ascend 910C devices |
  • Ref2VA migration changes
  • Opt-in BF16 prefix attention
| **291.25 → 286.67 s (−1.57%)**
5 s video, HTTP end-to-end | One lossless screening pair. Strict quality checks passed for the screened candidate. Overall campaign `INCONCLUSIVE`. | [Campaign #297](https://github.com/InferMatrixLab/InferRSI/issues/297) · [Evidence PR #317](https://github.com/InferMatrixLab/InferRSI/pull/317) | [#9745](https://github.com/vllm-project/vllm-omni/pull/9745) (draft) | -| **MiniMax-H3-Image** | 4× Ascend 910B1 |
  • Opt-in VAE tile graphs
  • Approximate encoder/decoder tile batching
| **16.89 → 10.82 s** · 4736×3552 generation
**4.18 → 2.67 s** · 1024×1024 generation
**5.68 → 4.57 s** · 1024×1024 editing | Single-launch HTTP medians vs. the same-layout SP4 control. All 30 numerical image screens passed. Latency targets were unmet and final qualification remains `INCONCLUSIVE`. | [Campaign #390](https://github.com/InferMatrixLab/InferRSI/issues/390) · [Evidence PR #394](https://github.com/InferMatrixLab/InferRSI/pull/394) | [#10184](https://github.com/vllm-project/vllm-omni/pull/10184) (open) | | **MiniMax-H3-Image** | 2× RTX 3090 |
  • Rank-local layerwise offload
  • Selected DiT weights kept resident
  • AdaLN caching
  • Request-local text refinement reuse
| **36.62 → 14.59 s (−60.16%)**
Warm mean HTTP delivery latency | One upstream A/B of complete deployment profiles. All nine image pairs had identical decoded RGB pixels, including warmups. Repeated qualification remains incomplete. | [Paired deployment benchmark](https://github.com/vllm-project/vllm-omni/pull/10243#:~:text=One%20paired%20deployment%20comparison) | [#10243](https://github.com/vllm-project/vllm-omni/pull/10243) (open) | -| **MiniCPM-o 4.5** | 1× L20X |
  • Packed conditioning on a tuned configuration
  • TensorRT speaker extraction + CUDA MPS (separate comparison)
| **+15.55%** throughput · **−27.69%** median TTFA
Packed conditioning, C32, same 50-frame window
**+20.5%** at C16 · **+25.4%** at C32
TensorRT + MPS throughput | Packed conditioning was a historical screen. The TensorRT + MPS comparison had speech quality verdict `FAIL`. | [Campaign #49](https://github.com/InferMatrixLab/InferRSI/issues/49) | [#9218](https://github.com/vllm-project/vllm-omni/pull/9218), [#9188](https://github.com/vllm-project/vllm-omni/pull/9188) (both merged) | -| **Qwen3-Omni** | 2× L20X |
  • cuDNN codec autotuning in the MRv2 deployment
| **+7.79%** throughput · **−6.06%** mean RTF
One C64 pilot | Median time to first audio **regressed by 7.12%**. Campaign `INCONCLUSIVE`. | [Campaign #112](https://github.com/InferMatrixLab/InferRSI/issues/112) · [Pinned pilot](https://github.com/InferMatrixLab/InferRSI/tree/e1ae256fc075ea9ab64aec68da749cf4a8c22ac7/evidence/qwen3-omni-2gpu-20261023/results/pr9283-cudnn-performance) | [#9283](https://github.com/vllm-project/vllm-omni/pull/9283) (draft) | | **LFM3-Omni-2B** | 1× Ascend 910B1 |
  • Depth graphs
  • Grouped greedy sampling
| **0.285 → 0.593 requests/s**
Upstream offline Omni API A/B, C4, temperature 0 | Retained audio waveforms matched bit-for-bit within each tested temperature. | [Offline A/B benchmark](https://github.com/vllm-project/vllm-omni/pull/9827#:~:text=Performance%20results) | [#9827](https://github.com/vllm-project/vllm-omni/pull/9827) (merged) | | **LingBot-World** | 4× Ascend 910C1 |
  • Whole-block paged KV gather with index_select
| **108.29 → 103.89 s (−4.06%)**
Full-request latency, upstream AB/BA | Pre-codec tensors and decoded RGB frames matched exactly. | [KV gather benchmark](https://github.com/vllm-project/vllm-omni/pull/8112#:~:text=Performance%3A%20whole%2Dblock%20index_select%20KV%20gather%20on%20NPU) | [#8112](https://github.com/vllm-project/vllm-omni/pull/8112) (merged) | -TTFA is time to first audio. RTF is real-time factor. - Upstream PR statuses checked on 2026-11-12.
From b454918039885f335e6db2ee5ec84b4b678ad219 Mon Sep 17 00:00:00 2001 From: zengchuang Date: Thu, 12 Nov 2026 20:16:31 +0800 Subject: [PATCH 11/11] docs: restore the full optimization results and evidence limits Agent-assisted-by: Codex (author) Signed-off-by: zengchuang --- README.md | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 57f495fce..a960c0974 100644 --- a/README.md +++ b/README.md @@ -65,7 +65,7 @@ For every CLI/Python operation — discovery, normalized reads, store candidates -## 📊 InferRSI × vLLM-Omni: Performance Highlights +## 📊 InferRSI × vLLM-Omni: Optimization Results InferRSI campaigns and upstream experiments cover image, video and speech models. Each result applies to its linked workload and measured revisions. Some results @@ -76,12 +76,29 @@ are reported directly by upstream contributors, with validation scoped to their | Model | Hardware | Optimization approach | Measured gains | Validation and limits | Evidence | vLLM-Omni PR | | --- | --- | --- | --- | --- | --- | --- | -| **Qwen-Image-2.1-Turbo** | 4× B300 |
  • Text-conditioning reuse
  • BF16 operator fusion
  • Symmetric-memory Ulysses with batched QKV exchange
| **384.16 → 266.60 ms (−30.60%)**
Warm HTTP end-to-end latency, repeated prompts | Historical InferRSI campaign. All 44 RGBA images were pixel-identical. | [Campaign #3](https://github.com/InferMatrixLab/InferRSI/issues/3) · [Pinned results](docs/qwen-image-2.1-turbo-b300.md) | [#8739](https://github.com/vllm-project/vllm-omni/pull/8739) (merged) | +| **Qwen-Image-2.1-Turbo** | 4× B300 |
  • Text-conditioning reuse
  • BF16 operator fusion
  • Symmetric-memory Ulysses with batched QKV exchange
| **384.16 → 266.60 ms (−30.60%)**
Warm HTTP end-to-end latency, repeated prompts | Historical InferRSI campaign. Latencies are medians of five trial means from alternating AB/BA groups. All 44 RGBA images were pixel-identical. | [Campaign #3](https://github.com/InferMatrixLab/InferRSI/issues/3) · [Pinned results](docs/qwen-image-2.1-turbo-b300.md) | [#8739](https://github.com/vllm-project/vllm-omni/pull/8739) (merged) | +| **Qwen-Image-2.1-Turbo** | 1× Ascend 910B1 |
  • Conditioning cache for repeated prompts
  • Shared modulation once per denoising step
  • Native interleaved NPU RoPE with frequency-table reuse
| **−22.20%** offline generation latency
Warm repeated prompts | Upstream parent-to-merge comparison. 18/18 RGBA images were pixel-identical. The 30% target was unmet. Both arms aborted during teardown, leaving lifecycle `INCONCLUSIVE`. | [Final-merge measurements](https://github.com/vllm-project/vllm-omni/pull/8729#:~:text=Final%2Dmerge%20performance%20and%20image%20comparison) · [Earlier Ascend cohorts](docs/qwen-image-2.1-turbo-ascend.md) | [#8729](https://github.com/vllm-project/vllm-omni/pull/8729) (merged) | | **MiniMax-H3 Ref2VA** | 8× Ascend 910B1 |
  • Lossy Cache-DiT profile tuning
| **407.27 → 108.12 s (−73.45%)**
5 s video, HTTP end-to-end vs. dense inference | All 14 frozen quality comparisons passed. | [Campaign #293](https://github.com/InferMatrixLab/InferRSI/issues/293) · [Pinned results](evidence/h3-ref2va-ascend-293/results/final-5s/README.md) | [#9698](https://github.com/vllm-project/vllm-omni/pull/9698) (draft) | +| **MiniMax-H3 Ref2VA** | 4 logical Ascend 910C devices |
  • Ref2VA migration changes
  • Opt-in BF16 prefix attention
| **291.25 → 286.67 s (−1.57%)**
5 s video, HTTP end-to-end | One lossless screening pair. Strict quality checks passed for the screened candidate. Overall campaign `INCONCLUSIVE`. | [Campaign #297](https://github.com/InferMatrixLab/InferRSI/issues/297) · [Evidence PR #317](https://github.com/InferMatrixLab/InferRSI/pull/317) | [#9745](https://github.com/vllm-project/vllm-omni/pull/9745) (draft) | +| **MiniMax-H3-Image** | 4× Ascend 910B1 |
  • Opt-in VAE tile graphs
  • Approximate encoder/decoder tile batching
| **16.89 → 10.82 s** · 4736×3552 generation
**4.18 → 2.67 s** · 1024×1024 generation
**5.68 → 4.57 s** · 1024×1024 editing | Single-launch HTTP medians vs. the same-layout SP4 control. All 30 numerical image screens passed. Latency targets were unmet and final qualification remains `INCONCLUSIVE`. | [Campaign #390](https://github.com/InferMatrixLab/InferRSI/issues/390) · [Evidence PR #394](https://github.com/InferMatrixLab/InferRSI/pull/394) | [#10184](https://github.com/vllm-project/vllm-omni/pull/10184) (open) | | **MiniMax-H3-Image** | 2× RTX 3090 |
  • Rank-local layerwise offload
  • Selected DiT weights kept resident
  • AdaLN caching
  • Request-local text refinement reuse
| **36.62 → 14.59 s (−60.16%)**
Warm mean HTTP delivery latency | One upstream A/B of complete deployment profiles. All nine image pairs had identical decoded RGB pixels, including warmups. Repeated qualification remains incomplete. | [Paired deployment benchmark](https://github.com/vllm-project/vllm-omni/pull/10243#:~:text=One%20paired%20deployment%20comparison) | [#10243](https://github.com/vllm-project/vllm-omni/pull/10243) (open) | +| **MiniCPM-o 4.5** | 1× L20X |
  • Packed conditioning on a tuned configuration
  • TensorRT speaker extraction + CUDA MPS (separate comparison)
| **+15.55%** throughput · **−27.69%** median TTFA
Packed conditioning, C32, same 50-frame window
**+20.5%** at C16 · **+25.4%** at C32
TensorRT + MPS throughput | The packed-conditioning candidate missed the exploratory WER margin against the tuning reference at the 50-frame setting. The separate TensorRT + MPS comparison had speech-quality verdict `FAIL`. | [Campaign #49](https://github.com/InferMatrixLab/InferRSI/issues/49) | [#9218](https://github.com/vllm-project/vllm-omni/pull/9218), [#9188](https://github.com/vllm-project/vllm-omni/pull/9188) (both merged) | +| **Qwen3-Omni** | 2× L20X |
  • cuDNN codec autotuning in the MRv2 deployment
| **+7.79%** throughput · **−6.06%** mean RTF
One C64 pilot | Median time to first audio **regressed by 7.12%**. Campaign `INCONCLUSIVE`. | [Campaign #112](https://github.com/InferMatrixLab/InferRSI/issues/112) · [Pinned pilot](https://github.com/InferMatrixLab/InferRSI/tree/e1ae256fc075ea9ab64aec68da749cf4a8c22ac7/evidence/qwen3-omni-2gpu-20261023/results/pr9283-cudnn-performance) | [#9283](https://github.com/vllm-project/vllm-omni/pull/9283) (draft) | | **LFM3-Omni-2B** | 1× Ascend 910B1 |
  • Depth graphs
  • Grouped greedy sampling
| **0.285 → 0.593 requests/s**
Upstream offline Omni API A/B, C4, temperature 0 | Retained audio waveforms matched bit-for-bit within each tested temperature. | [Offline A/B benchmark](https://github.com/vllm-project/vllm-omni/pull/9827#:~:text=Performance%20results) | [#9827](https://github.com/vllm-project/vllm-omni/pull/9827) (merged) | | **LingBot-World** | 4× Ascend 910C1 |
  • Whole-block paged KV gather with index_select
| **108.29 → 103.89 s (−4.06%)**
Full-request latency, upstream AB/BA | Pre-codec tensors and decoded RGB frames matched exactly. | [KV gather benchmark](https://github.com/vllm-project/vllm-omni/pull/8112#:~:text=Performance%3A%20whole%2Dblock%20index_select%20KV%20gather%20on%20NPU) | [#8112](https://github.com/vllm-project/vllm-omni/pull/8112) (merged) | +**Earlier Qwen-Image-2.1-Turbo Ascend measurements.** On one Ascend 910B1 at +1280×704, BF16, eager, and eight native steps, the author reported a 3111.5 ms +baseline. Conditioning reuse with interleaved RoPE reached **2459.6 ms (−21.0%)** +with pixel-identical RGBA. Adding the opt-in VAE fastpath reached **2425.5 ms +(−22.0%)** and passed the reported quality gate with changed pixels. Each mean +covers three requests after two warmups. InferRSI verification covered the +import, so acceptance remains `INCONCLUSIVE`. The 30% target was unmet. The B300 +campaign and this cohort used different hardware and measurement protocols. +[Earlier Ascend report](docs/qwen-image-2.1-turbo-ascend.md) + +TTFA is time to first audio. RTF is real-time factor. WER is word error rate. + Upstream PR statuses checked on 2026-11-12.