Skip to content

docs: InferRSI × vLLM-Omni Optimization Results - #415

Merged
zengchuang-hw merged 11 commits into
InferMatrixLab:mainfrom
zengchuang-hw:codex/readme-vllm-omni-results
Nov 12, 2026
Merged

zengchuang-hw merged 11 commits into
InferMatrixLab:mainfrom
zengchuang-hw:codex/readme-vllm-omni-results

Conversation

@zengchuang-hw

@zengchuang-hw zengchuang-hw commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Scope

Refs #316 for the documentation workstream.

The README presents 11 measured comparisons in 10 table rows across seven models and links 11 vLLM-Omni PRs. The table opens by default and shows the measurement scope, validation limits, and supporting evidence alongside each result.

This applies review option 1. The neutral Optimization Results heading covers all ten rows, including unmet targets, quality failures, and the Qwen3-Omni first-audio regression. A note below the table retains the earlier Qwen-Image Ascend 910B1 measurements separately from the final-merge comparison. Review

Verification

  • GitHub Markdown rendered the open-by-default table, all 10 rows, and four direct benchmark links. The relative evidence links resolve.
  • Rechecked the status of all 11 upstream PRs.
  • ruff check src tests tools, pytest tests/test_smoke.py tests/test_claim.py (8 passed), and git diff --check passed. The performance figures come from existing runs.

Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>

Copy link
Copy Markdown
Collaborator

Review at head a8bfeb629212544dde23a2a9ec85023b7440b787 (draft; base e2f4cd1e39b369f06835cdd655ad5dc9f6808918). This is a full review of the first revision, which changes only README.md (+21/−25). Not approving yet. The table is a good consolidation, and keeping the INCONCLUSIVE/FAIL limits in each row is the right call. I have a few traceability and wording points.

What I checked

  • All three relative links resolve at this head: docs/qwen-image-2.1-turbo-b300.md, docs/qwen-image-2.1-turbo-ascend.md and evidence/h3-ref2va-ascend-293/results/final-5s/README.md.
  • The upstream PR states in the last column match GitHub right now. Merged: #8739, #8729, #9218, #9188, #9827, #8112. Draft: #9698, #9745, #9283. Open: #10184, #10243.
  • I spot-checked figures against the upstream PR descriptions. #8729 reports 22.1959% (rounded to 22.20%), 18/18 pixel-identical, and all four processes exiting 134 at teardown. #8112 reports 108291 → 103890 ms (−4.06%) on 4× 910C1. The 910C1 measurement is correct, even though the #8112 title still says 910B1.

Please address

  1. README.md L75 (Qwen-Image Ascend row): the Evidence link doesn't support the numbers in the row. docs/qwen-image-2.1-turbo-ascend.md reports a different cohort: −22.0% vs a 3111.5 ms baseline, three requests after two warmups, author-reported. It doesn't contain the 22.20%, 18/18 or teardown-abort figures. Please link the measurement section of the #8729 description as the primary evidence and keep the doc as secondary. The old README and that doc both said the 30% target is not met. Please keep that note, or say why it no longer applies.
  2. L79, L82, L83 (H3-Image 2× RTX 3090, LFM3-Omni-2B, LingBot-World): "Benchmark and source pins in upstream PR" isn't a link. These three rows are reported by upstream contributors and have no InferRSI evidence. Please link the specific PR section or comment with the numbers, as row 1 does with pinned results, so readers can trace each figure.
  3. L81 (Qwen3-Omni): "7.12% higher median time to first audio" is a regression in a list of improvements. Please mark it explicitly, for example "median TTFA regressed 7.12%", so it isn't read as a gain.
  4. L66 heading ## InferRSI x vLLM-Omni联合优化. It's Chinese in an otherwise English README and has no space before 联合. Please use an English heading, such as ## InferRSI × vLLM-Omni optimization results. If a bilingual title is intended, please format it consistently.

Minor

  • The PR description says "10 measured comparisons". The MiniCPM-o row (L80) holds two separate comparisons, so it's 11 comparisons in 10 rows. Please adjust one or the other.
  • The merged/draft/open labels will go stale. An "as of 2026-11-12" note under the table would help.
  • L9: the badge now points to #results, which works because of the explicit <a id="results"> at L64. Please keep that anchor if the heading changes.

This is a static review. I didn't rerun any measurements.

Reviewed: e2f4cd1e39b369f06835cdd655ad5dc9f6808918...a8bfeb629212544dde23a2a9ec85023b7440b787 (full PR diff).

Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>
@Dong1017 Dong1017 added kind:docs Documentation and template delivery area:harness Harness module: profile, A/B loop, quality gates, evidence reports labels Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Addressed in 838fcb2, following your review:

  1. The Qwen-Image Ascend row now links the final-merge measurement section of [Perf][Qwen-Image-2.1] Reuse repeated conditioning and shared modulation vllm-project/vllm-omni#8729 as primary evidence, keeps the historical doc as secondary, and explicitly retains the unmet 30% target and teardown limitation.
  2. Added direct benchmark-section links for the H3-Image 2× RTX 3090, LFM3-Omni-2B and LingBot-World rows.
  3. Qwen3-Omni now explicitly says median time to first audio regressed by 7.12%.
  4. The heading is English-only. The existing #results anchor is unchanged.

Also dated the upstream PR statuses to 2026-11-12 and corrected the PR description to 11 comparisons in 10 rows.

Checks completed for this revision: GitHub Markdown rendering, source-heading and relative-link checks, ruff check src tests tools, pytest tests/test_smoke.py tests/test_claim.py (8 passed), and git diff --check. Draft CI passed for this head.

The performance figures remain results from existing runs; no performance experiments were rerun for this documentation update.

amy-why-3459
amy-why-3459 previously approved these changes Nov 12, 2026

@amy-why-3459 amy-why-3459 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base e2f4cd1e39b369f06835cdd655ad5dc9f6808918 and head 838fcb286fa5a2dc87f489578c146cb466cb575d. This is an incremental review of a8bfeb629212544dde23a2a9ec85023b7440b787..838fcb286fa5a2dc87f489578c146cb466cb575d (one commit, README.md +8/−6), checked against my previous review.

All points are resolved:

  1. L75: the Qwen-Image Ascend row now links the #8729 final-merge measurement section as primary evidence, with the doc as secondary, and keeps the unmet 30% target.
  2. The H3-Image 2× 3090, LFM3-Omni-2B and LingBot-World rows now have direct links. I checked that each text-fragment target heading exists in the upstream PR body: "Final-merge performance and image comparison" in #8729, "One paired deployment comparison" in #10243 (36.62/14.59/60.16 present), "Performance results" in #9827 (0.285/0.593 present), and "Performance: whole-block index_select KV gather on NPU" in #8112.
  3. Qwen3-Omni now states that median TTFA regressed 7.12%.
  4. The heading is English-only, and the #results anchor and badge target are unchanged.

The minor items are also done: statuses are dated 2026-11-12, and the description now says 11 comparisons in 10 rows.

CI at this head: declarations, full-check (3.10/3.11), installed-package (3.10/3.11) and pr-references all pass. This is a docs-only static review. I didn't rerun any measurements.

Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>

Copy link
Copy Markdown
Collaborator

增量 review:838fcb286fa5a2dc87f489578c146cb466cb575d..236b76c0ef19743e1f85432a447bb0c4a9fa77cf(1 个提交 docs: highlight standout vLLM-Omni performance gains,只改了 README.md 的 1 行)。

问题

  • README.md L66:标题从 InferRSI × vLLM-Omni optimization results 改成了 InferRSI × vLLM-Omni: Standout Performance Gains,和下面表格的内容对不上。表里有好几行算不上“standout gain”:

    • MiniMax-H3 Ref2VA · 910C 只有 −1.57%,整个 campaign 是 INCONCLUSIVE;
    • Qwen-Image-2.1-Turbo · 910B1 没达到 30% 目标,lifecycle 是 INCONCLUSIVE;
    • MiniMax-H3-Image · 910B1 没达到延迟目标,结论是 INCONCLUSIVE;
    • MiniCPM-o 4.5 的 TensorRT + MPS profile 语音质量是 FAIL;
    • Qwen3-Omni 的 TTFA 退化了 7.12%。

    这和 L69-70 “每条结果只适用于其链接的 workload 和 revision”的克制表述也有冲突。上一轮正是为了把这些限定条件写清楚才改的,现在的标题会让读者以为表里全是显著收益。

期望的修改:把标题改回中性的说法,比如 InferRSI × vLLM-Omni optimization results;或者从表里只挑出真正达标、而且有完整证据的条目放在 “Standout” 标题下面,其余条目单独列出并保留状态标注。#results 锚点是由 L64 的 <a id="results"> 提供的,改标题不会影响 L9、L11 的链接,这一点没问题。

我只看了这一行 diff 和 README 里的上下文,没有其他改动。这次是文档措辞,不涉及精度或性能实验。

Reviewed: 838fcb286fa5a2dc87f489578c146cb466cb575d → 236b76c0ef19743e1f85432a447bb0c4a9fa77cf.

Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>

Copy link
Copy Markdown
Collaborator

增量 review:236b76c0ef19743e1f85432a447bb0c4a9fa77cf..b2e2b0a994f63569818b40fee926ad010720dfdf(1 个提交 docs: separate optimization approaches and measured gains,README.md +12/−12)。这次仍不 approve,原因见下面“仍未关闭”。

新提交,已核对

  • 表格从 4 列拆成 7 列:Model / Hardware / Optimization approach / Measured gains / Validation and limits / Evidence / PR。所有数字、限定条件(INCONCLUSIVE、目标未达成、FAIL、TTFA 退化 7.12%)和链接都和 236b76c0 一致,没有丢失。

  • 新增的 “Optimization approach” 列,我抽查了上游 PR 的标题和正文,以下几条能对应上:

    • #8729:repeated conditioning / shared modulation;
    • #9745:opt-in BF16 prefix attention;
    • #10184:VAE graph;
    • #10243:weight transfers;
    • #9188:TensorRT speaker extraction;
    • #9283:codec autotuning;
    • #9827:正文里有 “depth graphs and grouped greedy sampling”;
    • #8112:正文里有 whole-block index_select KV gather,测量环境 “4x Ascend 910C1”,和 Hardware 列一致。

    其余几条细节(#8739 的 symmetric-memory Ulysses / batched QKV、#9698 的 Cache-DiT profile 调参)我没有逐字核对。

仍未关闭

  • README.md L66:标题还是 InferRSI × vLLM-Omni: Standout Performance Gains。这个提交的时间(07:36Z)早于我上一条评论,所以那条意见还没处理。表里有 −1.57%、三条 INCONCLUSIVE、一条 FAIL、一条 TTFA 退化,“Standout” 的说法和 L69-70 的限定表述冲突。请改回中性标题,比如 InferRSI × vLLM-Omni optimization results;或者只把达标的条目放在这个标题下面。

CI:b2e2b0a9 上的 full-check (3.10/3.11) 还在运行,其余 check 已通过。这次只是文档改动,不涉及精度或性能实验。

Reviewed: 236b76c0ef19743e1f85432a447bb0c4a9fa77cf → b2e2b0a994f63569818b40fee926ad010720dfdf.

Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>

Copy link
Copy Markdown
Contributor Author

Addressed the heading concern in your earlier review and follow-up with 72b3ca5d9296cebcf1216a803b42f32ade857791.

The title now reads “InferRSI × vLLM-Omni: Performance Highlights and Trade-offs”. This retains the performance highlights while making clear that the same table also includes incomplete, failed, and regressed outcomes, rather than presenting every row as a standout gain.

The commit changes only this heading. All table data, validation limits, evidence links, and the explicit #results anchor are unchanged.

Validation: tests/test_smoke.py and tests/test_claim.py: 8 passed; ruff check src tests tools and git diff --check passed. No performance measurements were rerun, and this does not claim a full-suite pass or change any campaign outcome. This is an author-side revision summary, not an independent review or approval.

amy-why-3459
amy-why-3459 previously approved these changes Nov 12, 2026

@amy-why-3459 amy-why-3459 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS

增量 review:b2e2b0a994f63569818b40fee926ad010720dfdf..72b3ca5d9296cebcf1216a803b42f32ade857791(1 个提交 docs: balance performance highlights with trade-offs,只改了 README.md L66 一行)。

  • 标题改成了 InferRSI × vLLM-Omni: Performance Highlights and Trade-offs,和表里既有显著收益、也有 INCONCLUSIVE / FAIL / TTFA 退化的内容对得上,我在 6817771750 和 6817937329 里提的标题问题已解决。
  • #results 锚点(L64)不受影响;表格内容和 b2e2b0a9 相同。
  • CI:72b3ca5d 上 full-check (3.10/3.11)、installed-package (3.10/3.11)、pr-references、declarations 全部通过。

文档改动,不涉及精度或性能实验。

Reviewed: b2e2b0a994f63569818b40fee926ad010720dfdf → 72b3ca5d9296cebcf1216a803b42f32ade857791.

Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>
@zengchuang-hw zengchuang-hw changed the title docs: showcase InferRSI x vLLM-Omni optimization results docs: InferRSI × vLLM-Omni Performance Highlights and Trade-offs Nov 12, 2026
Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>
amy-why-3459
amy-why-3459 previously approved these changes Nov 12, 2026

@amy-why-3459 amy-why-3459 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

增量 review:72b3ca5d9296cebcf1216a803b42f32ade857791..4cee566c0caf6c407d32053864e893ddba3bff33(2 个提交,仅 README.md +12/−10)。

  • 表格改为 “优化手段列表 + 数值在前、口径在后” 的格式,逐行核对了 9 行:所有数值、硬件、证据链接和上游 PR 状态都与上一版一致,只是把 “historical InferRSI campaign / upstream parent-to-merge comparison / same-layout SP4 control / one lossless screening pair” 等口径从 gains 列移到 validation 列,没有丢失归因。
  • RTX 3090 行由 “nine paired images” 改为 “nine image pairs”,与上游 vllm-project/vllm-omni#10243 的描述一致(每侧 9/9 请求,包括 3 次 warmup,九对输出 RGB 完全一致),表述更准确。
  • 新增的 TTFA / RTF 缩写说明合理;单元格里的 <ul><li> / <br> / <code> 在 GitHub 表格中可以正常渲染。

Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>
Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>
@zengchuang-hw zengchuang-hw changed the title docs: InferRSI × vLLM-Omni Performance Highlights and Trade-offs docs: InferRSI × vLLM-Omni Performance Highlights Nov 12, 2026
@zengchuang-hw

Copy link
Copy Markdown
Contributor Author

@amy-why-3459 @Dong1017 Plz cc and approve

@cursor

cursor Bot commented Nov 12, 2026

Copy link
Copy Markdown

Not approving.

@amy-why-3459's review of b2e2b0a994f63569818b40fee926ad010720dfdf still blocks on README.md L66. The heading InferRSI × vLLM-Omni: Standout Performance Gains sits over a mixed table: MiniMax-H3 Ref2VA · 910C at −1.57% (INCONCLUSIVE), Qwen-Image-2.1-Turbo · 910B1 and MiniMax-H3-Image · 910B1 (INCONCLUSIVE, targets unmet), MiniCPM-o 4.5 TensorRT + MPS (FAIL), and Qwen3-Omni median TTFA regressed 7.12%. That conflicts with the limits in L69–70.

72b3ca5d9296cebcf1216a803b42f32ade857791 renames the heading to “InferRSI × vLLM-Omni: Performance Highlights and Trade-offs”. “Highlights” still frames those rows as standout results, and “Trade-offs” mislabels INCONCLUSIVE and FAIL. The open request is a neutral heading such as InferRSI × vLLM-Omni optimization results, or a split that keeps only fully evidenced, gate-meeting rows under a highlights heading.

The thread has no independent re-read of 72b3ca5d. Dong-bot will re-read after that heading or split is on the head. This cc leaves @amy-why-3459's finding open.

Copy link
Copy Markdown
Collaborator

增量 review:4cee566c0caf6c407d32053864e893ddba3bff33..290b1e6db058184acd460bba621a7b2040796ffd(2 个提交,只改了 README.md,+12/−3)。这次先不 approve,原因见下面的“问题”。

已核对,没问题

  • 给 🧭 Choose your starting path(L54)和 🧪 Standalone engine benchmarks(L96)显式加了 <a id>,加 emoji 后自动生成的 slug 变了也不影响跳转。#results(L66)没动,L9 的徽章和 L11 的导航链接都还有效。
  • <details> / <summary> 前后都留了空行(L74–L75、L94),表格放在折叠块里也能正常渲染。写法和 Architecture 那节(L240–L252)一致。
  • 表格的行、数值、口径和证据链接跟 4cee566c 完全一样,没有内容丢失。

问题

  1. README.md L68:标题从 Performance Highlights and Trade-offs 改回了 Performance Highlights。 加上 “and Trade-offs” 是为了解决我在 6817771750 和 6817937329 里提的问题(72b3ca5d 当时也是这么说明的)。表里仍然有 −1.57%、三条 INCONCLUSIVE、一条 FAIL 和 TTFA 退化 7.12%,只写 “Highlights” 又会让人觉得表里全是收益。另外,PR 标题现在还是 “…Performance Highlights and Trade-offs”,和 README 不一致。
  2. L74–L75:表格默认折叠后,这个问题更明显。 读者不展开的话,只能看到 “Performance Highlights” 标题和 L70–L72 的简介,看不到任何限定条件。

期望的修改:L68 改回 📊 InferRSI × vLLM-Omni: Performance Highlights and Trade-offs,或者换一个同样中性的标题。如果想保留折叠,可以在 L75 的 summary 里点明内容包括限制,比如 View benchmarks, limits and upstream PRs。如果标题确实要缩短,请顺便同步 PR 标题,并说明为什么这样不会再引起之前的误读。

这次只是文档改动,不涉及精度或性能实验。

Reviewed: 4cee566c0caf6c407d32053864e893ddba3bff33 → 290b1e6db058184acd460bba621a7b2040796ffd.

@zengchuang-hw
zengchuang-hw marked this pull request as ready for review November 12, 2026 11:40
Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR review verdict — DO NOT PASS

Reviewed base e2f4cd1e39b369f06835cdd655ad5dc9f6808918 and head 72bdd6c3ea5c0dda35cc05f1b72a89ba296ed454. Full diff versus main (no previous-head SHA seeded). The change is README.md only: badge target, the Choose-your-starting-path anchor, and the #results section.

Incremental check against the dismissed reviews on this PR (4cee566c, 72b3ca5d, 838fcb28):

  • Still in place: H3-Image, LFM3-Omni-2B, and LingBot-World text-fragment links; explicit #results anchor; English heading; badge target #results.
  • Not on this head: the Qwen-Image Ascend row (unmet 30% target and #8729 link), the Qwen3-Omni median TTFA regression, and the “Performance Highlights and Trade-offs” heading.

Blockers:

  1. README.md L68–L87 replaces the results section and removes the Ascend NPU Qwen-Image cohort that main publishes in this section: 2459.6 ms (−21.0%, pixel-identical RGBA), 2425.5 ms (−22.0%, quality gate passed, not pixel-identical), the author-reported and not-independently-verified limit, “The 30% target is not met.”, the B300-vs-NPU protocol split, and the links to docs/qwen-image-2.1-turbo-ascend.md and vllm-omni#8729. The new table (L79–L83) has no Qwen-Image Ascend row, while L79 still leads with 384.16 → 266.60 ms (−30.60%). The dismissed review of 838fcb28 required this row to stay, with the #8729 final-merge section as the primary evidence and the unmet 30% target kept. Restore that row (numbers, limits, and both links) in this table.

  2. README.md L68 is again InferRSI × vLLM-Omni: Performance Highlights, and the L79–L83 “Measured gains” cells are only improvements. The 72b3ca5d review required the heading to mention trade-offs because this section also carried INCONCLUSIVE / FAIL rows and a TTFA regression. The 838fcb28 review required Qwen3-Omni to state that median TTFA regressed 7.12%. That wording is absent from 72bdd6c3. Restore those non-positive rows, including the Qwen3-Omni TTFA regression, and make the heading match the table.

Optional:

  • The B300 cell (L79) no longer says the figure is the median of trial means from alternating AB/BA groups. That protocol was in the paragraph this hunk deletes. The campaign doc link is still there; putting the protocol back in the gains or validation cell would keep 266.60 ms from being read as a single run.
  • Explicit anchors for choose-your-starting-path and standalone-engine-benchmarks match the new emoji headings.

Checked in the diff: the five deltas match the stated percents (30.60%, 73.45%, 60.16%, 4.06%; LFM3 is a throughput pair with no percent). The H3-Image row still says nine image pairs, identical decoded RGB, and incomplete repeated qualification.

Not verified here: live upstream PR status for the “checked on 2026-11-12” line, the linked campaign docs, evidence/h3-ref2va-ascend-293/results/final-5s/README.md, or CI on 72bdd6c3. Docs-only change; tests were not run.

This review does not approve the PR. I will re-read after the author pushes.

Copy link
Copy Markdown
Collaborator

增量 review:290b1e6db058184acd460bba621a7b2040796ffd..72bdd6c3ea5c0dda35cc05f1b72a89ba296ed454(1 个提交 docs: narrow performance highlights and expand the table by default,README.md +2/−9)。这次仍不 approve。

已核对

  • L74–L75 改成了 <details open>,summary 也改成 Benchmarks, validation limits, and upstream PRs,我上一条评论第 2 点(默认折叠)已经解决。
  • 留下的 5 行(L79–L83:Qwen-Image B300、H3 Ref2VA 910B1、H3-Image 2×3090、LFM3-Omni-2B、LingBot-World)和 290b1e6d 逐字一致,数值、口径和链接都没变。

问题

  1. L79–L83:为了配合 “Highlights” 标题,直接删掉了 5 行结果,不是把它们移到别处。 删掉的是 Qwen-Image-2.1-Turbo · 910B1(−22.20%,30% 目标未达成,INCONCLUSIVE)、H3 Ref2VA · 910C(−1.57%,INCONCLUSIVE)、H3-Image · 4×910B1(目标未达成,INCONCLUSIVE)、MiniCPM-o 4.5(TensorRT + MPS 语音质量 FAIL)和 Qwen3-Omni(TTFA 退化 7.12%)。我之前提过两个方案:一是用中性标题;二是只把达标的条目放在 highlights 下面,其余条目单独列出并保留状态标注。现在的改法属于第二种,但少了“单独列出”这一步,README 里只剩正面结果,读者看不到这些未达标和退化的情况。
  2. 这次改动会把 main 上已经公开的 Ascend 结果删掉。 main 的 README L84–L89 写着 Qwen-Image Ascend 的两组数据:2459.6 ms(−21.0%,pixel-identical)和 2425.5 ms(−22.0%,未 pixel-identical)。同一处还有 “The 30% target is not met.”、B300 和 NPU 测量协议不同的说明,以及 docs/qwen-image-2.1-turbo-ascend.md 和 vllm-omni#8729 的链接。现在这个 head 上,结果区已经完全没有 Ascend Qwen-Image 的内容。cursor[bot] 在这个 head 上给的 DO NOT PASS 也指出了这一点,我核对过 main,属实。
  3. PR 描述和标题已经对不上了。 描述还写着 “11 measured comparisons in 10 rows across seven models, with 11 corresponding vLLM-Omni PRs”,但表格现在只有 5 行、5 个 PR;PR 标题也还是 “…and Trade-offs”。

期望的修改(二选一)

  • 恢复这 5 行,标题改成中性的说法,比如 InferRSI × vLLM-Omni: Optimization Results,或者改回 Performance Highlights and Trade-offs;
  • 或者保留现在的 highlights 表,在下面另加一个小节(比如 Other measured results and limits),把这 5 行连同数值、INCONCLUSIVE / FAIL / 退化说明和证据链接原样放进去。Qwen-Image Ascend 那行要保留 #8729 final-merge 的链接和 “30% target unmet” 的说明。

不管选哪种,都请同步更新 PR 标题和描述里的行数和 PR 数。

这次只是文档改动,不涉及精度或性能实验。

Reviewed: 290b1e6db058184acd460bba621a7b2040796ffd → 72bdd6c3ea5c0dda35cc05f1b72a89ba296ed454.

Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>
@zengchuang-hw zengchuang-hw changed the title docs: InferRSI × vLLM-Omni Performance Highlights docs: InferRSI × vLLM-Omni Optimization Results Nov 12, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base e2f4cd1e39b369f06835cdd655ad5dc9f6808918 and head b454918039885f335e6db2ee5ec84b4b678ad219.
Incremental range 72bdd6c3ea5c0dda35cc05f1b72a89ba296ed454..b454918039885f335e6db2ee5ec84b4b678ad219 (README.md only). No formal APPROVED review on this head. Amy's Approves on 838fcb28, 72b3ca5d, and 4cee566c are dismissed older heads. No outstanding CHANGES_REQUESTED.

Prior DO NOT PASS on 72bdd6c3 is resolved:

  1. Ascend Qwen-Image is back, split so the final-merge result and the earlier cohort are not collapsed. The new table row is 1× Ascend 910B1, −22.20% offline generation latency, 18/18 RGBA pixel-identical, 30% target unmet, both arms aborted in teardown so lifecycle INCONCLUSIVE. Primary evidence is the #8729 fragment Final-merge performance and image comparison; secondary is docs/qwen-image-2.1-turbo-ascend.md. The paragraph under the table restores the earlier cohort: author-reported 3111.5 ms baseline, 2459.6 ms (−21.0%) pixel-identical RGBA, 2425.5 ms (−22.0%) quality gate with changed pixels, three requests after two warmups, import-only verification so acceptance INCONCLUSIVE, 30% unmet, and the B300-vs-NPU protocol split, with the same doc link. Those two −22% figures are different runs (22.0% of 3111.5 ms is 2425.5 ms; −22.20% is not that ratio).
  2. The heading is no longer Performance Highlights. It is Optimization Results. Limits sit in the table (unmet 30%, unmet H3-Image latency targets, Ref2VA 910C INCONCLUSIVE, MiniCPM-o WER miss and speech-quality FAIL, Qwen3-Omni TTFA regression), so the highlights-only title regression is gone.
  3. Qwen3-Omni is a row again: +7.79% throughput, −6.06% mean RTF, median TTFA regressed by 7.12%, campaign INCONCLUSIVE, pinned at e1ae256fc075ea9ab64aec68da749cf4a8c22ac7.

Also checked in this diff: B300 still states 44 pixel-identical RGBA images and now says medians of five trial means from alternating AB/BA groups. Percentage arithmetic on the rows that quote both endpoints matches the stated percents (B300 −30.60%, Ref2VA −73.45% and −1.57%, H3-Image 3090 −60.16%, LingBot −4.06%, earlier Ascend −21.0% / −22.0%). #results anchor is unchanged. Previously checked fragments on #10243, #9827, and #8112 are unchanged. H3 Ref2VA 910C, H3-Image 910B1, and MiniCPM-o keep the miss or FAIL / INCONCLUSIVE in the validation column next to the gains.

Not verified here: measurements were not rerun. Upstream PR bodies, campaign issues #49 / #112 / #297 / #390, evidence PRs #317 / #394, and CI on b4549180 were not re-fetched. Docs-only static review of the incremental diff.

@amy-why-3459 amy-why-3459 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

增量 review:72bdd6c3ea5c0dda35cc05f1b72a89ba296ed454..b454918039885f335e6db2ee5ec84b4b678ad219(1 个提交 docs: restore the full optimization results and evidence limits,只改了 README.md)。

我上一条评论里的三个问题都已解决:

  • L68 标题改成了中性的 InferRSI × vLLM-Omni: Optimization Results,对应方案一。
  • L80、L82、L83、L85、L86 恢复了被删掉的 5 行。Qwen-Image Ascend、H3 Ref2VA 910C、H3-Image 4×910B1、Qwen3-Omni 这 4 行和 290b1e6d 逐字一致,都保留了 INCONCLUSIVE、30% 目标未达成和 TTFA 退化 7.12% 的说明;Ascend 这一行也保留了 #8729 final-merge 的链接。
  • L90–L98 把 main 上的 Ascend 数据(3111.5 → 2459.6 ms / 2425.5 ms,−21.0% / −22.0%)和 B300 与 NPU 测量协议不同的说明加了回来,和 docs/qwen-image-2.1-turbo-ascend.md 一致。
  • PR 标题和描述已经同步:10 行、11 组对比、7 个模型、11 个上游 PR,我逐一数过。

这次的新写法也核对过:

  • L79 B300 新加的 “medians of five trial means from alternating AB/BA groups”,和 main L74 以及 docs/qwen-image-2.1-turbo-b300.md 的说法一致。
  • L85 MiniCPM-o 新写的 “missed the exploratory WER margin … at the 50-frame setting”,和 vllm-omni#9218 的 exploratory 语音质量表一致(50-frame 在 C1/C16/C32 都超过了 WER 均值的 margin)。

这次只是文档改动,不涉及精度或性能实验。当前 head 上 cursor[bot] 已经 APPROVED,没有其他阻塞意见。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:harness Harness module: profile, A/B loop, quality gates, evidence reports kind:docs Documentation and template delivery

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants