You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
task: add GPU measurement-validity and profiler-attribution guidance to inference-benchmark-design #455
Thanks, the scope is well bounded: documentation only, status stays candidate / empirical not_run, and it is complementary to #454 without a hard dependency. The associated draft is #456 at 8c0b894a2dc9b8572700ca3f80b119e75f2377db. A few points before acceptance:
Owner and acceptance. The issue is still Proposed, and the goal-acceptance owner is TBD. Please have that owner recorded before treating the slice as accepted.
Matrix placement. The new docs/reference-matrix.md row in knowledge: add GPU measurement-validity and profiler attribution guidance #456 sits under "Task-specific benchmark selection (issue [RFC] Benchmark selection by task for InferRSI #60)", so the generated docs/skill-coverage.md also lists it as an [RFC] Benchmark selection by task for InferRSI #60 source, and the searchspace note was inserted under the same heading. This guidance is general GPU measurement methodology, not a task-selection source. Please move the matrix row and the searchspace paragraph to a dedicated short section (similar to the later "Faster CosyVoice evaluation source context" block) so the generated coverage label is accurate.
Adoption wording. "Require an explanation before adoption" should remain clearly candidate guidance, not a repository-wide gate, until an owner decision or rfc: 可解释的改进——用 profile 证据和配对统计判定 PASS #454 says otherwise. The SKILL.md pointer already calls it candidate guidance; please keep that if the paragraph is edited.
Author-reported numbers. The four task-supplied figures are clearly labelled unverified and not reusable thresholds; please keep them that way.
One source I checked. Triton v3.2.0 is 9641643da6c52000c807b5eeed05edaec4402a67, and do_bench does a write-based cache.zero_() flush, includes that flush in the preliminary estimate, and places per-sample events only around fn. That matches the reference. I did not re-download the NVIDIA/PyTorch documents to re-check the retained hashes.
#456 is also based on a92c77ccaed68426c83154cff7619745f34d0058; current main is cd8fcb452bd2c7969a7a6e2544aa1cb5f8ad5332 (#453), which added C1 assertions to tests/check_installed.py while leaving the resource count at 923. Please rebase before merge and regenerate records if the source-revision commit changes. I'll comment on the PR separately for the matrix section and the rebase.
Keep “require an explanation before adoption” clearly candidate guidance, not a repository-wide gate, until an owner decision or rfc: 可解释的改进——用 profile 证据和配对统计判定 PASS #454 says otherwise.
This acceptance authorizes the scoped knowledge work. It is not Approve of #456 and not Issue closure.
Activity
amy-why-3459 commentedon Nov 15, 2026
Thanks, the scope is well bounded: documentation only, status stays candidate / empirical
not_run, and it is complementary to #454 without a hard dependency. The associated draft is #456 at8c0b894a2dc9b8572700ca3f80b119e75f2377db. A few points before acceptance:docs/reference-matrix.mdrow in knowledge: add GPU measurement-validity and profiler attribution guidance #456 sits under "Task-specific benchmark selection (issue [RFC] Benchmark selection by task for InferRSI #60)", so the generateddocs/skill-coverage.mdalso lists it as an [RFC] Benchmark selection by task for InferRSI #60 source, and the searchspace note was inserted under the same heading. This guidance is general GPU measurement methodology, not a task-selection source. Please move the matrix row and the searchspace paragraph to a dedicated short section (similar to the later "Faster CosyVoice evaluation source context" block) so the generated coverage label is accurate.9641643da6c52000c807b5eeed05edaec4402a67, anddo_benchdoes a write-basedcache.zero_()flush, includes that flush in the preliminary estimate, and places per-sample events only aroundfn. That matches the reference. I did not re-download the NVIDIA/PyTorch documents to re-check the retained hashes.#456 is also based on
a92c77ccaed68426c83154cff7619745f34d0058; current main iscd8fcb452bd2c7969a7a6e2544aa1cb5f8ad5332(#453), which added C1 assertions totests/check_installed.pywhile leaving the resource count at 923. Please rebase before merge and regenerate records if the source-revision commit changes. I'll comment on the PR separately for the matrix section and the rebase.Dong1017 commentedon Nov 15, 2026
@BeatSeat — steward decision for @Dong1017 (authorized Dong-bot / InferRSI on-call delegate; comment posted as Dong1017).
Decision: accept scope (with conditions) for this documentation-only knowledge slice.
8c0b894a2dc9b8572700ca3f80b119e75f2377dbAccepted
methods/inference-benchmark-designplus a short SKILL.md pointer.not_run/ efficacynot_evaluatedpreserved; no new skill, runner, numeric gate, GPU execution, or performance claim.Conditions before ready-for-review / Approve (also recorded on #456)
main(cd8fcb45/ Package shared technical contracts and retire root modules #453) and refresh declarationhead_sha/ projections if the source-revision commit changes.This acceptance authorizes the scoped knowledge work. It is not Approve of #456 and not Issue closure.
zengchuang-hw commentedon Nov 17, 2026
独立需求检视(基于 main
1bbc3b0c8de14d6194d7ee7987607365b665fbfe,不是实现验收):当前 owner 仍是
candidate / empirical_validation=not_run,已有固定工作量、统计口径、共享机器比较及 trial identity 指导;本次读取的 main tree 尚无references/gpu-measurement-validity.md。建议以补充测量证据契约落地,不新增 runner 或通用数值门槛。接受范围前,建议明确以下边界:
“同 run 归因”不要等同于把 profiler 时间当最终 A/B 时间。 建议分别保存无 profiler 的计时 run、对应的归因 capture、候选/基线 artifact identity、输入/shape/dtype、设备与软件栈、warmup、stream/graph、采样范围以及 replay/cache/clock 配置。匹配的是可追溯的实验条件与调用区间;若 profiler 改变执行方式,要显式声明差异,不以 capture 文件同名证明等价。NVIDIA 当前 Profiling Guide 明确区分常规与插桩执行及 metric collection overhead;此处的当前文档核查不是对要求中的 2025.3 快照完成核验。
Amdahl 是带假设的一致性检查,不是自动拒绝规则。 固定工作量、其余成本不变且可加的串行分解,才可用 baseline fraction 与局部加速比推导整体预期。不能直接把多 stream 的重叠 kernel duration 总和当端到端占比。超出预期时应复核计时边界、重叠、融合、工作量变化及隐藏同步,而不是直接判为造假或性能 FAIL。CUDA 12.9.1 的强扩展/Amdahl 与异步 CPU/GPU 计时章节支持保留这些前提。
把计时有效性、机制解释和最终接受分开。 工作量 counters 要写清单位、分母、聚合区间及是否来自 replay。降低 kernel 次数可能是合法融合,不必要求各项 counter 完全相等;但删工作、改变输出语义或质量缺失不能只靠更快的计时通过。缺少机制证据时应限制因果归因,不把“未解释”偷换为已测延迟不存在,也不把计时改善直接升级为服务收益。
metadata 与 matrix 有一个具体冲突风险。 当前 owner 的
searchspace_not_applicable明确把硬件身份/隔离视为比较控制而非资源机制。若新增内容仍是测量指导,应沿用 supplementary Resource 导航;若拟新增正向资源层映射,须同时审视并修订 N/A 理由,不能同时保留冲突的两种声明。knowledge-base contract 明确要求区分这一点。验收应覆盖内容进入真实消费路径。 新 reference 加入 owner
provenance_readsets;先提交源内容,再按tools/build_skill_index.py --source-revision <committed-source-revision>重生并以同 revision 执行--check。不要手改 generated record。当前 installed checker 对完整 inventory 固定计数且支持 checkout 字节比对,应按实际新增文件更新 fixture,并从隔离安装确认该 reference 可读;仅搜索到一个标题不足以验收。建议的轻量内容回归案例:异步 launch-only CPU 计时、profiled 与 unprofiled 时间混用、重叠 kernel 总和误作 wall-time 占比、合法融合导致 counter 变化、缺失原始记录的收益自述。每例给出所缺证据与允许/不允许的结论;无需新增 GPU 执行范围。
已执行:读取当前 main 的 AGENTS、协作/知识契约、owner SKILL/metadata/reference、生成器与 installed checker;核对 Issue 类型/作者并检查本人既有评论去重;读取 CUDA 12.9.1 官方计时/Amdahl 章节及当前 Nsight Compute 指导。2025.3 archive 本次访问失败,其余列举版本/论文与原始收益数据未全面复核。未执行:代码修改、生成器、pytest、wheel 安装、GPU/profiler 测量或性能复现。因此上述是接受前建议,不是声称实现完成、性能 PASS、自动接受范围或实施承诺。
一手参考:CUDA 12.9.1 Best Practices、当前 Nsight Compute Profiling Guide。