Repository navigation
docs: InferRSI × vLLM-Omni Optimization Results - #415
zengchuang-hw merged 11 commits into
Conversation
Agent-assisted-by: Codex (author) Signed-off-by: zengchuang <zengchuang3@huawei.com>
|
Review at head What I checked
Please address
Minor
This is a static review. I didn't rerun any measurements. Reviewed: |
Agent-assisted-by: Codex (author) Signed-off-by: zengchuang <zengchuang3@huawei.com>
|
Addressed in 838fcb2, following your review:
Also dated the upstream PR statuses to 2026-11-12 and corrected the PR description to 11 comparisons in 10 rows. Checks completed for this revision: GitHub Markdown rendering, source-heading and relative-link checks, The performance figures remain results from existing runs; no performance experiments were rerun for this documentation update. |
amy-why-3459
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base e2f4cd1e39b369f06835cdd655ad5dc9f6808918 and head 838fcb286fa5a2dc87f489578c146cb466cb575d. This is an incremental review of a8bfeb629212544dde23a2a9ec85023b7440b787..838fcb286fa5a2dc87f489578c146cb466cb575d (one commit, README.md +8/−6), checked against my previous review.
All points are resolved:
- L75: the Qwen-Image Ascend row now links the #8729 final-merge measurement section as primary evidence, with the doc as secondary, and keeps the unmet 30% target.
- The H3-Image 2× 3090, LFM3-Omni-2B and LingBot-World rows now have direct links. I checked that each text-fragment target heading exists in the upstream PR body: "Final-merge performance and image comparison" in #8729, "One paired deployment comparison" in #10243 (36.62/14.59/60.16 present), "Performance results" in #9827 (0.285/0.593 present), and "Performance: whole-block index_select KV gather on NPU" in #8112.
- Qwen3-Omni now states that median TTFA regressed 7.12%.
- The heading is English-only, and the
#resultsanchor and badge target are unchanged.
The minor items are also done: statuses are dated 2026-11-12, and the description now says 11 comparisons in 10 rows.
CI at this head: declarations, full-check (3.10/3.11), installed-package (3.10/3.11) and pr-references all pass. This is a docs-only static review. I didn't rerun any measurements.
Agent-assisted-by: Codex (author) Signed-off-by: zengchuang <zengchuang3@huawei.com>
|
增量 review: 问题
期望的修改:把标题改回中性的说法,比如 我只看了这一行 diff 和 README 里的上下文,没有其他改动。这次是文档措辞,不涉及精度或性能实验。 Reviewed: |
Agent-assisted-by: Codex (author) Signed-off-by: zengchuang <zengchuang3@huawei.com>
|
增量 review: 新提交,已核对
仍未关闭
CI: Reviewed: |
Agent-assisted-by: Codex (author) Signed-off-by: zengchuang <zengchuang3@huawei.com>
|
Addressed the heading concern in your earlier review and follow-up with 72b3ca5d9296cebcf1216a803b42f32ade857791. The title now reads “InferRSI × vLLM-Omni: Performance Highlights and Trade-offs”. This retains the performance highlights while making clear that the same table also includes incomplete, failed, and regressed outcomes, rather than presenting every row as a standout gain. The commit changes only this heading. All table data, validation limits, evidence links, and the explicit Validation: |
amy-why-3459
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
增量 review:b2e2b0a994f63569818b40fee926ad010720dfdf..72b3ca5d9296cebcf1216a803b42f32ade857791(1 个提交 docs: balance performance highlights with trade-offs,只改了 README.md L66 一行)。
- 标题改成了
InferRSI × vLLM-Omni: Performance Highlights and Trade-offs,和表里既有显著收益、也有INCONCLUSIVE/FAIL/ TTFA 退化的内容对得上,我在 6817771750 和 6817937329 里提的标题问题已解决。 #results锚点(L64)不受影响;表格内容和b2e2b0a9相同。- CI:
72b3ca5d上 full-check (3.10/3.11)、installed-package (3.10/3.11)、pr-references、declarations 全部通过。
文档改动,不涉及精度或性能实验。
Reviewed: b2e2b0a994f63569818b40fee926ad010720dfdf → 72b3ca5d9296cebcf1216a803b42f32ade857791.
Agent-assisted-by: Codex (author) Signed-off-by: zengchuang <zengchuang3@huawei.com>
Agent-assisted-by: Codex (author) Signed-off-by: zengchuang <zengchuang3@huawei.com>
amy-why-3459
left a comment
There was a problem hiding this comment.
LGTM
增量 review:72b3ca5d9296cebcf1216a803b42f32ade857791..4cee566c0caf6c407d32053864e893ddba3bff33(2 个提交,仅 README.md +12/−10)。
- 表格改为 “优化手段列表 + 数值在前、口径在后” 的格式,逐行核对了 9 行:所有数值、硬件、证据链接和上游 PR 状态都与上一版一致,只是把 “historical InferRSI campaign / upstream parent-to-merge comparison / same-layout SP4 control / one lossless screening pair” 等口径从 gains 列移到 validation 列,没有丢失归因。
- RTX 3090 行由 “nine paired images” 改为 “nine image pairs”,与上游 vllm-project/vllm-omni#10243 的描述一致(每侧 9/9 请求,包括 3 次 warmup,九对输出 RGB 完全一致),表述更准确。
- 新增的 TTFA / RTF 缩写说明合理;单元格里的
<ul><li>/<br>/<code>在 GitHub 表格中可以正常渲染。
Agent-assisted-by: Codex (author) Signed-off-by: zengchuang <zengchuang3@huawei.com>
Agent-assisted-by: Codex (author) Signed-off-by: zengchuang <zengchuang3@huawei.com>
|
@amy-why-3459 @Dong1017 Plz cc and approve |
|
Not approving. @amy-why-3459's review of
The thread has no independent re-read of |
|
增量 review: 已核对,没问题
问题
期望的修改:L68 改回 这次只是文档改动,不涉及精度或性能实验。 Reviewed: |
Agent-assisted-by: Codex (author) Signed-off-by: zengchuang <zengchuang3@huawei.com>
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Reviewed base e2f4cd1e39b369f06835cdd655ad5dc9f6808918 and head 72bdd6c3ea5c0dda35cc05f1b72a89ba296ed454. Full diff versus main (no previous-head SHA seeded). The change is README.md only: badge target, the Choose-your-starting-path anchor, and the #results section.
Incremental check against the dismissed reviews on this PR (4cee566c, 72b3ca5d, 838fcb28):
- Still in place: H3-Image, LFM3-Omni-2B, and LingBot-World text-fragment links; explicit
#resultsanchor; English heading; badge target#results. - Not on this head: the Qwen-Image Ascend row (unmet 30% target and #8729 link), the Qwen3-Omni median TTFA regression, and the “Performance Highlights and Trade-offs” heading.
Blockers:
-
README.mdL68–L87 replaces the results section and removes the Ascend NPU Qwen-Image cohort that main publishes in this section: 2459.6 ms (−21.0%, pixel-identical RGBA), 2425.5 ms (−22.0%, quality gate passed, not pixel-identical), the author-reported and not-independently-verified limit, “The 30% target is not met.”, the B300-vs-NPU protocol split, and the links todocs/qwen-image-2.1-turbo-ascend.mdand vllm-omni#8729. The new table (L79–L83) has no Qwen-Image Ascend row, while L79 still leads with 384.16 → 266.60 ms (−30.60%). The dismissed review of838fcb28required this row to stay, with the #8729 final-merge section as the primary evidence and the unmet 30% target kept. Restore that row (numbers, limits, and both links) in this table. -
README.mdL68 is againInferRSI × vLLM-Omni: Performance Highlights, and the L79–L83 “Measured gains” cells are only improvements. The72b3ca5dreview required the heading to mention trade-offs because this section also carriedINCONCLUSIVE/FAILrows and a TTFA regression. The838fcb28review required Qwen3-Omni to state that median TTFA regressed 7.12%. That wording is absent from72bdd6c3. Restore those non-positive rows, including the Qwen3-Omni TTFA regression, and make the heading match the table.
Optional:
- The B300 cell (L79) no longer says the figure is the median of trial means from alternating AB/BA groups. That protocol was in the paragraph this hunk deletes. The campaign doc link is still there; putting the protocol back in the gains or validation cell would keep 266.60 ms from being read as a single run.
- Explicit anchors for
choose-your-starting-pathandstandalone-engine-benchmarksmatch the new emoji headings.
Checked in the diff: the five deltas match the stated percents (30.60%, 73.45%, 60.16%, 4.06%; LFM3 is a throughput pair with no percent). The H3-Image row still says nine image pairs, identical decoded RGB, and incomplete repeated qualification.
Not verified here: live upstream PR status for the “checked on 2026-11-12” line, the linked campaign docs, evidence/h3-ref2va-ascend-293/results/final-5s/README.md, or CI on 72bdd6c3. Docs-only change; tests were not run.
This review does not approve the PR. I will re-read after the author pushes.
|
增量 review: 已核对
问题
期望的修改(二选一)
不管选哪种,都请同步更新 PR 标题和描述里的行数和 PR 数。 这次只是文档改动,不涉及精度或性能实验。 Reviewed: |
Agent-assisted-by: Codex (author) Signed-off-by: zengchuang <zengchuang3@huawei.com>
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base e2f4cd1e39b369f06835cdd655ad5dc9f6808918 and head b454918039885f335e6db2ee5ec84b4b678ad219.
Incremental range 72bdd6c3ea5c0dda35cc05f1b72a89ba296ed454..b454918039885f335e6db2ee5ec84b4b678ad219 (README.md only). No formal APPROVED review on this head. Amy's Approves on 838fcb28, 72b3ca5d, and 4cee566c are dismissed older heads. No outstanding CHANGES_REQUESTED.
Prior DO NOT PASS on 72bdd6c3 is resolved:
- Ascend Qwen-Image is back, split so the final-merge result and the earlier cohort are not collapsed. The new table row is 1× Ascend 910B1, −22.20% offline generation latency, 18/18 RGBA pixel-identical, 30% target unmet, both arms aborted in teardown so lifecycle
INCONCLUSIVE. Primary evidence is the #8729 fragmentFinal-merge performance and image comparison; secondary isdocs/qwen-image-2.1-turbo-ascend.md. The paragraph under the table restores the earlier cohort: author-reported 3111.5 ms baseline, 2459.6 ms (−21.0%) pixel-identical RGBA, 2425.5 ms (−22.0%) quality gate with changed pixels, three requests after two warmups, import-only verification so acceptanceINCONCLUSIVE, 30% unmet, and the B300-vs-NPU protocol split, with the same doc link. Those two −22% figures are different runs (22.0% of 3111.5 ms is 2425.5 ms; −22.20% is not that ratio). - The heading is no longer
Performance Highlights. It isOptimization Results. Limits sit in the table (unmet 30%, unmet H3-Image latency targets, Ref2VA 910CINCONCLUSIVE, MiniCPM-o WER miss and speech-qualityFAIL, Qwen3-Omni TTFA regression), so the highlights-only title regression is gone. - Qwen3-Omni is a row again: +7.79% throughput, −6.06% mean RTF, median TTFA regressed by 7.12%, campaign
INCONCLUSIVE, pinned ate1ae256fc075ea9ab64aec68da749cf4a8c22ac7.
Also checked in this diff: B300 still states 44 pixel-identical RGBA images and now says medians of five trial means from alternating AB/BA groups. Percentage arithmetic on the rows that quote both endpoints matches the stated percents (B300 −30.60%, Ref2VA −73.45% and −1.57%, H3-Image 3090 −60.16%, LingBot −4.06%, earlier Ascend −21.0% / −22.0%). #results anchor is unchanged. Previously checked fragments on #10243, #9827, and #8112 are unchanged. H3 Ref2VA 910C, H3-Image 910B1, and MiniCPM-o keep the miss or FAIL / INCONCLUSIVE in the validation column next to the gains.
Not verified here: measurements were not rerun. Upstream PR bodies, campaign issues #49 / #112 / #297 / #390, evidence PRs #317 / #394, and CI on b4549180 were not re-fetched. Docs-only static review of the incremental diff.
amy-why-3459
left a comment
There was a problem hiding this comment.
LGTM
增量 review:72bdd6c3ea5c0dda35cc05f1b72a89ba296ed454..b454918039885f335e6db2ee5ec84b4b678ad219(1 个提交 docs: restore the full optimization results and evidence limits,只改了 README.md)。
我上一条评论里的三个问题都已解决:
- L68 标题改成了中性的
InferRSI × vLLM-Omni: Optimization Results,对应方案一。 - L80、L82、L83、L85、L86 恢复了被删掉的 5 行。Qwen-Image Ascend、H3 Ref2VA 910C、H3-Image 4×910B1、Qwen3-Omni 这 4 行和
290b1e6d逐字一致,都保留了INCONCLUSIVE、30% 目标未达成和 TTFA 退化 7.12% 的说明;Ascend 这一行也保留了 #8729 final-merge 的链接。 - L90–L98 把 main 上的 Ascend 数据(3111.5 → 2459.6 ms / 2425.5 ms,−21.0% / −22.0%)和 B300 与 NPU 测量协议不同的说明加了回来,和
docs/qwen-image-2.1-turbo-ascend.md一致。 - PR 标题和描述已经同步:10 行、11 组对比、7 个模型、11 个上游 PR,我逐一数过。
这次的新写法也核对过:
- L79 B300 新加的 “medians of five trial means from alternating AB/BA groups”,和 main L74 以及
docs/qwen-image-2.1-turbo-b300.md的说法一致。 - L85 MiniCPM-o 新写的 “missed the exploratory WER margin … at the 50-frame setting”,和 vllm-omni#9218 的 exploratory 语音质量表一致(50-frame 在 C1/C16/C32 都超过了 WER 均值的 margin)。
这次只是文档改动,不涉及精度或性能实验。当前 head 上 cursor[bot] 已经 APPROVED,没有其他阻塞意见。
Scope
Refs #316 for the documentation workstream.
The README presents 11 measured comparisons in 10 table rows across seven models and links 11 vLLM-Omni PRs. The table opens by default and shows the measurement scope, validation limits, and supporting evidence alongside each result.
This applies review option 1. The neutral Optimization Results heading covers all ten rows, including unmet targets, quality failures, and the Qwen3-Omni first-audio regression. A note below the table retains the earlier Qwen-Image Ascend 910B1 measurements separately from the final-merge comparison. Review
Verification
ruff check src tests tools,pytest tests/test_smoke.py tests/test_claim.py(8 passed), andgit diff --checkpassed. The performance figures come from existing runs.