Skip to content

task: add GPU measurement-validity and profiler-attribution guidance to inference-benchmark-design #455

Open

Description

@BeatSeat

Current state: 提案。已在 fork 上准备好对应的知识 PR(只有文档,不改代码)。
Decision needed: 请 owner 确认是否接受这个有边界的知识补充,并指定验收负责人。
Next action: 评审关联 PR;PR 中的作者声明和生成记录都已按 knowledge-model policy 准备好。

Goal and acceptance

  • Problem / desired outcome: GPU 端到端计时会随时钟、温度、功耗和共享负载漂移;而 profiler replay(例如 Nsight Compute 默认锁定 base clock 并清缓存)的运行条件又和正常运行不同。仓库现在缺少一份说明:怎样记录可比的测量,以及怎样把收益归因到具体机制。
  • Completion criteria: 在 methods/inference-benchmark-design 下新增一份 owner-local reference,并在 SKILL.md 中加一段简短指引。内容包括:时钟和缓存控制、同一次运行内的归因与 Amdahl 一致性检查、工作量计数、"先解释后采纳"的流程,以及常见计时陷阱。所有内容要么引用固定版本的公开来源,要么明确标注为作者自述、未验证。保持 candidate、empirical not_run 的状态,生成记录按两步流程重新生成。

Scope and constraints

  • In scope / out of scope:
    • 包括:reference 文件、owner metadata、两张 matrix 的导航行、生成记录、安装资源计数 fixture。
    • 不包括:新 skill、benchmark runner、数值门槛、GPU 实测、性能声明。
  • Relevant dependencies, shared interfaces or limits: 与 rfc: 可解释的改进——用 profile 证据和配对统计判定 PASS #454(可解释改进的 Harness 支持)互补,两者不互相依赖。文中四条作者自述的数字没有提供原始数据,只作为背景,不作为证据。

Ownership and authorization

  • Proposed by / actual requester: BeatSeat
  • Status: Proposed
  • Accountable parent (Issue-level owner, if assigned): TBD
  • Slice implementer(s) / acting contributor (if assigned): BeatSeat
  • Goal-acceptance owner: TBD
  • Accepted decision or scoped delegation: Not yet granted

Inputs and deliverables

  • Relevant sources, revisions or linked issues: Nsight Compute / Nsight Systems 2025.3、CUDA 12.9.1 与 10.2 文档、按哈希固定的 nvidia-smi 文档、Triton v3.2.0、PyTorch 2.7、robust-kbench (arXiv:2509.14279v1)。具体检查过的章节记录在 owner metadata 中。
  • Expected artifacts: references/gpu-measurement-validity.md(175 行)、SKILL.md 中 6 行指引、metadata 与 matrix 更新、重新生成的记录。

Completion and closure

  • Who accepts completion and may close: 指定的验收负责人。
  • Remaining uncertainty / next decision: 这份内容只是候选指南,后续是否需要在设备上做实证评估,另行决定。

Activity

  1. amy-why-3459 commented on Nov 15, 2026

    @amy-why-3459
    Collaborator

    Thanks, the scope is well bounded: documentation only, status stays candidate / empirical not_run, and it is complementary to #454 without a hard dependency. The associated draft is #456 at 8c0b894a2dc9b8572700ca3f80b119e75f2377db. A few points before acceptance:

    1. Owner and acceptance. The issue is still Proposed, and the goal-acceptance owner is TBD. Please have that owner recorded before treating the slice as accepted.
    2. Matrix placement. The new docs/reference-matrix.md row in knowledge: add GPU measurement-validity and profiler attribution guidance #456 sits under "Task-specific benchmark selection (issue [RFC] Benchmark selection by task for InferRSI #60)", so the generated docs/skill-coverage.md also lists it as an [RFC] Benchmark selection by task for InferRSI #60 source, and the searchspace note was inserted under the same heading. This guidance is general GPU measurement methodology, not a task-selection source. Please move the matrix row and the searchspace paragraph to a dedicated short section (similar to the later "Faster CosyVoice evaluation source context" block) so the generated coverage label is accurate.
    3. Adoption wording. "Require an explanation before adoption" should remain clearly candidate guidance, not a repository-wide gate, until an owner decision or rfc: 可解释的改进——用 profile 证据和配对统计判定 PASS #454 says otherwise. The SKILL.md pointer already calls it candidate guidance; please keep that if the paragraph is edited.
    4. Author-reported numbers. The four task-supplied figures are clearly labelled unverified and not reusable thresholds; please keep them that way.
    5. One source I checked. Triton v3.2.0 is 9641643da6c52000c807b5eeed05edaec4402a67, and do_bench does a write-based cache.zero_() flush, includes that flush in the preliminary estimate, and places per-sample events only around fn. That matches the reference. I did not re-download the NVIDIA/PyTorch documents to re-check the retained hashes.

    #456 is also based on a92c77ccaed68426c83154cff7619745f34d0058; current main is cd8fcb452bd2c7969a7a6e2544aa1cb5f8ad5332 (#453), which added C1 assertions to tests/check_installed.py while leaving the resource count at 923. Please rebase before merge and regenerate records if the source-revision commit changes. I'll comment on the PR separately for the matrix section and the rebase.

  2. Dong1017 commented on Nov 15, 2026

    @Dong1017
    Collaborator

    @BeatSeat — steward decision for @Dong1017 (authorized Dong-bot / InferRSI on-call delegate; comment posted as Dong1017).

    Decision: accept scope (with conditions) for this documentation-only knowledge slice.

    Field Record
    Status Proposed → Accepted (knowledge contribution scope)
    Accountable parent / goal-acceptance owner Dong1017
    Slice implementer BeatSeat
    Associated draft #456 @ 8c0b894a2dc9b8572700ca3f80b119e75f2377db

    Accepted

    Conditions before ready-for-review / Approve (also recorded on #456)

    1. Move the reference-matrix row and searchspace note out of the issue [RFC] Benchmark selection by task for InferRSI #60 section into a dedicated short section, then regenerate coverage/records.
    2. Rebase onto current main (cd8fcb45 / Package shared technical contracts and retire root modules #453) and refresh declaration head_sha / projections if the source-revision commit changes.
    3. Keep “require an explanation before adoption” clearly candidate guidance, not a repository-wide gate, until an owner decision or rfc: 可解释的改进——用 profile 证据和配对统计判定 PASS #454 says otherwise.

    This acceptance authorizes the scoped knowledge work. It is not Approve of #456 and not Issue closure.

  3. added
    area:knowledgeKnowledge line: packs, skill library, knowledge admission
    kind:taskBounded delivery task with a delegated slice
    on Nov 16, 2026
  4. zengchuang-hw commented on Nov 17, 2026

    @zengchuang-hw
    Contributor

    独立需求检视(基于 main 1bbc3b0c8de14d6194d7ee7987607365b665fbfe,不是实现验收):

    当前 owner 仍是 candidate / empirical_validation=not_run,已有固定工作量、统计口径、共享机器比较及 trial identity 指导;本次读取的 main tree 尚无 references/gpu-measurement-validity.md。建议以补充测量证据契约落地,不新增 runner 或通用数值门槛。

    接受范围前,建议明确以下边界:

    1. “同 run 归因”不要等同于把 profiler 时间当最终 A/B 时间。 建议分别保存无 profiler 的计时 run、对应的归因 capture、候选/基线 artifact identity、输入/shape/dtype、设备与软件栈、warmup、stream/graph、采样范围以及 replay/cache/clock 配置。匹配的是可追溯的实验条件与调用区间;若 profiler 改变执行方式,要显式声明差异,不以 capture 文件同名证明等价。NVIDIA 当前 Profiling Guide 明确区分常规与插桩执行及 metric collection overhead;此处的当前文档核查不是对要求中的 2025.3 快照完成核验。

    2. Amdahl 是带假设的一致性检查,不是自动拒绝规则。 固定工作量、其余成本不变且可加的串行分解,才可用 baseline fraction 与局部加速比推导整体预期。不能直接把多 stream 的重叠 kernel duration 总和当端到端占比。超出预期时应复核计时边界、重叠、融合、工作量变化及隐藏同步,而不是直接判为造假或性能 FAIL。CUDA 12.9.1 的强扩展/Amdahl 与异步 CPU/GPU 计时章节支持保留这些前提。

    3. 把计时有效性、机制解释和最终接受分开。 工作量 counters 要写清单位、分母、聚合区间及是否来自 replay。降低 kernel 次数可能是合法融合,不必要求各项 counter 完全相等;但删工作、改变输出语义或质量缺失不能只靠更快的计时通过。缺少机制证据时应限制因果归因,不把“未解释”偷换为已测延迟不存在,也不把计时改善直接升级为服务收益。

    4. metadata 与 matrix 有一个具体冲突风险。 当前 owner 的 searchspace_not_applicable 明确把硬件身份/隔离视为比较控制而非资源机制。若新增内容仍是测量指导,应沿用 supplementary Resource 导航;若拟新增正向资源层映射,须同时审视并修订 N/A 理由,不能同时保留冲突的两种声明。knowledge-base contract 明确要求区分这一点。

    5. 验收应覆盖内容进入真实消费路径。 新 reference 加入 owner provenance_readsets;先提交源内容,再按 tools/build_skill_index.py --source-revision <committed-source-revision> 重生并以同 revision 执行 --check。不要手改 generated record。当前 installed checker 对完整 inventory 固定计数且支持 checkout 字节比对,应按实际新增文件更新 fixture,并从隔离安装确认该 reference 可读;仅搜索到一个标题不足以验收。

    建议的轻量内容回归案例:异步 launch-only CPU 计时、profiled 与 unprofiled 时间混用、重叠 kernel 总和误作 wall-time 占比、合法融合导致 counter 变化、缺失原始记录的收益自述。每例给出所缺证据与允许/不允许的结论;无需新增 GPU 执行范围。

    已执行:读取当前 main 的 AGENTS、协作/知识契约、owner SKILL/metadata/reference、生成器与 installed checker;核对 Issue 类型/作者并检查本人既有评论去重;读取 CUDA 12.9.1 官方计时/Amdahl 章节及当前 Nsight Compute 指导。2025.3 archive 本次访问失败,其余列举版本/论文与原始收益数据未全面复核。未执行:代码修改、生成器、pytest、wheel 安装、GPU/profiler 测量或性能复现。因此上述是接受前建议,不是声称实现完成、性能 PASS、自动接受范围或实施承诺。

    一手参考:CUDA 12.9.1 Best Practices、当前 Nsight Compute Profiling Guide。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:knowledgeKnowledge line: packs, skill library, knowledge admissionkind:taskBounded delivery task with a delegated slice

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions