diff --git a/README.md b/README.md index 6f1893956..a960c0974 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ [![Python 3.10+](https://img.shields.io/badge/Python-3.10%2B-3776AB?style=flat-square)](pyproject.toml) [![CLI](https://img.shields.io/badge/Interface-CLI-14B8A6?style=flat-square)](docs/quickstart.md) -[![vLLM-Omni PR](https://img.shields.io/badge/vLLM--Omni-Upstream_PR-8B5CF6?style=flat-square)](https://github.com/vllm-project/vllm-omni/pull/8739) +[![vLLM-Omni contributions](https://img.shields.io/badge/vLLM--Omni-Contributions-8B5CF6?style=flat-square)](#results) [Why InferRSI](#why-inferrsi) · [How It Works](#how-it-works) · [Results](#results) · [Quickstart](#quickstart) · [Architecture](#architecture) · [Docs](#docs) @@ -51,7 +51,9 @@ The **offline harness-improvement MVP** adds a separate Skill Eval path for supp declarative candidates. Its fixed-output experiments validate the controller; real-model gains remain unmeasured. [Flow, limits and reproduction →](docs/harness-improvement.md) -## Choose your starting path + + +## 🧭 Choose your starting path | You want to… | Start at | | --- | --- | @@ -63,30 +65,47 @@ For every CLI/Python operation — discovery, normalized reads, store candidates -## 📊 From profiling to a 30.60% latency reduction - -**Qwen-Image-2.1-Turbo · 4× NVIDIA B300 · 1280×704 · BF16 · Native 8 steps** - -| Baseline E2E | Optimized E2E | Image validation | -| :---: | :---: | :---: | -| **384.16 ms** | **266.60 ms** | **44/44 pixel-identical** | +## 📊 InferRSI × vLLM-Omni: Optimization Results -Recorded local campaign: five alternating AB/BA groups, warmed repeated prompts with new seeds, and complete RGBA PNG responses. Accuracy checks cover 40 formal images and 4 held-out images. Latencies are medians of trial means for this workload. +InferRSI campaigns and upstream experiments cover image, video and speech models. +Each result applies to its linked workload and measured revisions. Some results +are reported directly by upstream contributors, with validation scoped to their benchmarks. -[Results, source pins & every iteration →](docs/qwen-image-2.1-turbo-b300.md) · [Acceptance issue](https://github.com/InferMatrixLab/InferRSI/issues/3) · [Upstream PR](https://github.com/vllm-project/vllm-omni/pull/8739) +
+Benchmarks, validation limits, and upstream PRs -### Ascend NPU: up to 22.0% reported latency reduction - -**Qwen-Image-2.1-Turbo · 1× Ascend 910B1 · 1280×704 · BF16 · Native 8 steps · eager** +| Model | Hardware | Optimization approach | Measured gains | Validation and limits | Evidence | vLLM-Omni PR | +| --- | --- | --- | --- | --- | --- | --- | +| **Qwen-Image-2.1-Turbo** | 4× B300 | | **384.16 → 266.60 ms (−30.60%)**
Warm HTTP end-to-end latency, repeated prompts | Historical InferRSI campaign. Latencies are medians of five trial means from alternating AB/BA groups. All 44 RGBA images were pixel-identical. | [Campaign #3](https://github.com/InferMatrixLab/InferRSI/issues/3) · [Pinned results](docs/qwen-image-2.1-turbo-b300.md) | [#8739](https://github.com/vllm-project/vllm-omni/pull/8739) (merged) | +| **Qwen-Image-2.1-Turbo** | 1× Ascend 910B1 | | **−22.20%** offline generation latency
Warm repeated prompts | Upstream parent-to-merge comparison. 18/18 RGBA images were pixel-identical. The 30% target was unmet. Both arms aborted during teardown, leaving lifecycle `INCONCLUSIVE`. | [Final-merge measurements](https://github.com/vllm-project/vllm-omni/pull/8729#:~:text=Final%2Dmerge%20performance%20and%20image%20comparison) · [Earlier Ascend cohorts](docs/qwen-image-2.1-turbo-ascend.md) | [#8729](https://github.com/vllm-project/vllm-omni/pull/8729) (merged) | +| **MiniMax-H3 Ref2VA** | 8× Ascend 910B1 | | **407.27 → 108.12 s (−73.45%)**
5 s video, HTTP end-to-end vs. dense inference | All 14 frozen quality comparisons passed. | [Campaign #293](https://github.com/InferMatrixLab/InferRSI/issues/293) · [Pinned results](evidence/h3-ref2va-ascend-293/results/final-5s/README.md) | [#9698](https://github.com/vllm-project/vllm-omni/pull/9698) (draft) | +| **MiniMax-H3 Ref2VA** | 4 logical Ascend 910C devices | | **291.25 → 286.67 s (−1.57%)**
5 s video, HTTP end-to-end | One lossless screening pair. Strict quality checks passed for the screened candidate. Overall campaign `INCONCLUSIVE`. | [Campaign #297](https://github.com/InferMatrixLab/InferRSI/issues/297) · [Evidence PR #317](https://github.com/InferMatrixLab/InferRSI/pull/317) | [#9745](https://github.com/vllm-project/vllm-omni/pull/9745) (draft) | +| **MiniMax-H3-Image** | 4× Ascend 910B1 | | **16.89 → 10.82 s** · 4736×3552 generation
**4.18 → 2.67 s** · 1024×1024 generation
**5.68 → 4.57 s** · 1024×1024 editing | Single-launch HTTP medians vs. the same-layout SP4 control. All 30 numerical image screens passed. Latency targets were unmet and final qualification remains `INCONCLUSIVE`. | [Campaign #390](https://github.com/InferMatrixLab/InferRSI/issues/390) · [Evidence PR #394](https://github.com/InferMatrixLab/InferRSI/pull/394) | [#10184](https://github.com/vllm-project/vllm-omni/pull/10184) (open) | +| **MiniMax-H3-Image** | 2× RTX 3090 | | **36.62 → 14.59 s (−60.16%)**
Warm mean HTTP delivery latency | One upstream A/B of complete deployment profiles. All nine image pairs had identical decoded RGB pixels, including warmups. Repeated qualification remains incomplete. | [Paired deployment benchmark](https://github.com/vllm-project/vllm-omni/pull/10243#:~:text=One%20paired%20deployment%20comparison) | [#10243](https://github.com/vllm-project/vllm-omni/pull/10243) (open) | +| **MiniCPM-o 4.5** | 1× L20X | | **+15.55%** throughput · **−27.69%** median TTFA
Packed conditioning, C32, same 50-frame window
**+20.5%** at C16 · **+25.4%** at C32
TensorRT + MPS throughput | The packed-conditioning candidate missed the exploratory WER margin against the tuning reference at the 50-frame setting. The separate TensorRT + MPS comparison had speech-quality verdict `FAIL`. | [Campaign #49](https://github.com/InferMatrixLab/InferRSI/issues/49) | [#9218](https://github.com/vllm-project/vllm-omni/pull/9218), [#9188](https://github.com/vllm-project/vllm-omni/pull/9188) (both merged) | +| **Qwen3-Omni** | 2× L20X | | **+7.79%** throughput · **−6.06%** mean RTF
One C64 pilot | Median time to first audio **regressed by 7.12%**. Campaign `INCONCLUSIVE`. | [Campaign #112](https://github.com/InferMatrixLab/InferRSI/issues/112) · [Pinned pilot](https://github.com/InferMatrixLab/InferRSI/tree/e1ae256fc075ea9ab64aec68da749cf4a8c22ac7/evidence/qwen3-omni-2gpu-20261023/results/pr9283-cudnn-performance) | [#9283](https://github.com/vllm-project/vllm-omni/pull/9283) (draft) | +| **LFM3-Omni-2B** | 1× Ascend 910B1 | | **0.285 → 0.593 requests/s**
Upstream offline Omni API A/B, C4, temperature 0 | Retained audio waveforms matched bit-for-bit within each tested temperature. | [Offline A/B benchmark](https://github.com/vllm-project/vllm-omni/pull/9827#:~:text=Performance%20results) | [#9827](https://github.com/vllm-project/vllm-omni/pull/9827) (merged) | +| **LingBot-World** | 4× Ascend 910C1 | | **108.29 → 103.89 s (−4.06%)**
Full-request latency, upstream AB/BA | Pre-codec tensors and decoded RGB frames matched exactly. | [KV gather benchmark](https://github.com/vllm-project/vllm-omni/pull/8112#:~:text=Performance%3A%20whole%2Dblock%20index_select%20KV%20gather%20on%20NPU) | [#8112](https://github.com/vllm-project/vllm-omni/pull/8112) (merged) | + +**Earlier Qwen-Image-2.1-Turbo Ascend measurements.** On one Ascend 910B1 at +1280×704, BF16, eager, and eight native steps, the author reported a 3111.5 ms +baseline. Conditioning reuse with interleaved RoPE reached **2459.6 ms (−21.0%)** +with pixel-identical RGBA. Adding the opt-in VAE fastpath reached **2425.5 ms +(−22.0%)** and passed the reported quality gate with changed pixels. Each mean +covers three requests after two warmups. InferRSI verification covered the +import, so acceptance remains `INCONCLUSIVE`. The 30% target was unmet. The B300 +campaign and this cohort used different hardware and measurement protocols. +[Earlier Ascend report](docs/qwen-image-2.1-turbo-ascend.md) + +TTFA is time to first audio. RTF is real-time factor. WER is word error rate. + +Upstream PR statuses checked on 2026-11-12. -| Optimization | Mean vs 3111.5 ms baseline | Reported image validation | -| --- | :---: | --- | -| Conditioning reuse + interleaved RoPE | **2459.6 ms (−21.0%)** | Pixel-identical RGBA | -| + opt-in VAE fastpath | **2425.5 ms (−22.0%)** | Quality gate passed; not pixel-identical | +
-Author-reported means from three requests after two warmups; not independently verified by this import. **The 30% target is not met.** The B300 and NPU cohorts use separate hardware and measurement protocols. + -[Ascend measurements and rejected rounds →](docs/qwen-image-2.1-turbo-ascend.md) · [Upstream NPU patch](https://github.com/vllm-project/vllm-omni/pull/8729) +## 🧪 Standalone engine benchmarks ### Standalone video/audio engines