Skip to content

harness: explain gains with profile evidence and paired statistics - #457

Draft
BeatSeat wants to merge 11 commits into
InferMatrixLab:mainfrom
BeatSeat:feat/paired-measurement-statistics
Draft

BeatSeat wants to merge 11 commits into
InferMatrixLab:mainfrom
BeatSeat:feat/paired-measurement-statistics

Conversation

@BeatSeat

@BeatSeat BeatSeat commented Nov 15, 2026 •

Copy link
Copy Markdown

Governing Issue and delivery slice

Refs #454

  • Accepted scope / delegation: rfc: 可解释的改进——用 profile 证据和配对统计判定 PASS #454 已由 steward 代 Dong1017 接受方向(附条件):可选能力、两个开关默认关闭、按一个 PR 评审;硬件 A/A 不是合入前提(决定)。本 head 处理了全部条件和四位评审的意见(含 linyueqian 复核后提出的重复次数下限,以及 david6666666 指出的配对模式选优问题),见"评审后的修订"。
  • What this PR delivers: 普通项目优化(campaign start/round/final)新增三项能力,默认都不改变现有判定:
    1. 配对统计(默认只报告):按 steps() 已有的 AB/BA 交替,把同一次重复里的基线和候选样本配对,报告逐对加速比中位数的精确次序统计量区间(符号检验反演,至少 6 对),以及区间与门槛的关系。
    2. 用 profile 解释改进:测量脚本可以额外输出 breakdown(分区域耗时)和 counters(工作量计数);每轮可以用 --mechanism-file 声明机制和预期方向;Harness 按声明条数做 Bonferroni 校正后机械核对,给出每条核对的 outcome 和 explanation.status。
    3. 两个冻结开关:--require-explanation 只采纳 explained 的候选;--decision paired 用配对区间判定,并按逐对加速比中位数选优(未配对中位数不能否决配对 PASS);区间跨门槛判 INCONCLUSIVE(within_noise),不被采纳也不阻塞搜索。两者同时打开时,先由 --decision 得出性能结论,--require-explanation 只降级 PASS。
  • What it does not deliver: 默认判定变更;基准题包支持(benchmarks session 拒绝 --mechanism-file);Dashboard 展示;与具体 profiler 绑定的采集脚本;GPU/NPU 实测。

Knowledge authorship declaration

本 PR 不触及 knowledge-model policy 的受保护路径,预期为 Not applicable。

{
  "schema_version": 1,
  "head_sha": "507ce154bc80e68e0a303e26af8a4db79f4313d4",
  "human_only": false,
  "material_ai_authors": [
    {
      "tool": "Claude Code",
      "model_id": "claude-opus-5-5",
      "effort": "unknown",
      "role": "author"
    }
  ]
}

Changed artifacts and evidence

文件 改动 作用
src/inferrsi/harness/measurement.py 新增 359 行 精确次序统计量区间(interval_rank、median_interval、min_pairs,二项尾用有理数精确比较)、配对统计、profile_evidence(Bonferroni)(原始与归一化方向相反时记为 raw_normalized_conflict)、validate_mechanism、explain;无法计算的值一律返回 unavailable,记录保持严格 JSON
src/inferrsi/harness/project.py +99/−7 assess() 接入两个开关和机制核对;--decision paired 时 score/speedup/improvement_pct 取逐对加速比中位数(基线 1.0),点估计保留在 point_estimate,不可表示的值记为空而不阻断;profile 报告出错时记为 unavailable,不影响判定;新增设置项的校验(--require-explanation 至少 6 次重复)
src/inferrsi/harness/optimization.py +48/−8 在执行前把声明和摘要写入 pending;settle 时核对摘要,并把内存中的声明传给判定器;final 使用被选中轮次绑定的声明;每轮记录 selection(adopted / not_eligible / not_better_than_best / baseline_identity);明确 eligible: false 的完成轮次可以没有 score
src/inferrsi/harness/optimization_session.py +30/−8 冻结两个开关;paired 时把选优冻结为 paired_median_speedup(maximize,初值 1.0);读取并校验 --mechanism-file,开关打开时在执行前拒绝所需对数超过冻结重复次数的声明;_runtime_identity 纳入 measurement.py
src/inferrsi/apps/views/dashboard_data.py +6/−2 受管 campaign 行用测得指标(metrics.name/direction)标注中位数,不用选优指标
src/inferrsi/apps/cli/arguments.py +10 --decision、--require-explanation、--mechanism-file
docs/optimization-workflow-unification.md +40 中文说明:字段、取值、精确区间与 Bonferroni、开关与优先级、paired 下的选优与 selection 记录、局限
tests/test_harness_measurement.py 新增 91 个用例(含参数化) 精确秩(n = 1…40 逐一核对最大性)、最少对数与 Bonferroni 所需对数、门槛处 A/A 误报不超过精确上界、上千对不溢出、方向判定、零基线、参照区域不互证、unchanged 只记"未推翻"、原始与归一化方向相反不被读成任一方向、机制校验与分类
tests/test_project_explanation.py 新增 29 个用例 端到端:配对 PASS 按配对加速比被采纳并由 final 复测(评审者的 6 vs 9 情形)、更快的早期候选使后来的 PASS 记为 not_better_than_best;配对统计不可用或点估计溢出时记录轮次而不阻断;确认的机制在 final 复核;不可采纳情形都让搜索继续;开关关闭时只报告;within_noise 为 INCONCLUSIVE;两开关组合 6 例;点估计未过而区间过门槛判 PASS;profile 报告出错不改判;重复次数不足时 start 拒绝、声明无法核对时在执行前拒绝且不耗预算、开关关闭时仍只报告;篡改 state.json 或副本;中断后伪造声明的恢复
tests/test_project_optimization.py、tests/test_optimization_session.py +4、+2 个用例 配对统计不改判、不可达门槛、旧记录可读;题包会话拒绝机制声明;无 score 的不可采纳轮次正常结算、final 不报加速比
tools/measurement_simulation.py 新增 126 行 可复算的合成消融;--gate-pct 0 给出门槛处 A/A 比例与精确上界

判定逻辑

explanation.status 含义 --require-explanation 下
explained 每条预期变化都 confirmed,且没有任何一条 contradicted 可采纳
unsupported 有预期变化 not_shown(未检出;区间仍可能包含预期的变化) INCONCLUSIVE,继续搜索
contradicted 方向相反,或声明 unchanged 的项检出变化 FAIL,继续搜索
unverifiable 缺少数据、比值无定义、原始与归一化方向相反、对数不足以支撑校正后的置信度(k = 1 需 6 对,2–3 需 7 对,4–6 需 8 对),或证据按更少条数校正 INCONCLUSIVE,继续搜索
not_declared 本轮没有声明机制 INCONCLUSIVE,继续搜索
--decision paired 区间与门槛 verdict / reason
above_gate PASS / paired_interval_above_gate(即使点估计未过门槛;此时 agrees_with_point_gate: false)
straddles_gate INCONCLUSIVE / within_noise(没能证明过门槛,不是门槛未达到的证据)
below_gate、gate_unreachable FAIL / performance_gate_not_met
统计不可用 INCONCLUSIVE / paired_statistics_unavailable,不退回未配对规则

在 --require-explanation 下只有 contradicted(数据与预期相反)记 FAIL,证据不足的情形都记 INCONCLUSIVE,与配对性能判定的原则一致。若某区域原始比较和"除以参照区域"后的比较给出明确相反的方向(例如原始增加 20%、相对参照却下降),记为 raw_normalized_conflict,该条核对为 unverifiable:机制声明无法表达"只预期相对变化",两个方向都不采信。

声明为 unchanged 的项只会被记为 not_contradicted(未检出变化)或 contradicted,不会被"确认不变":没有等价性容差时,区间包含 1 不是"没变"的证据。声明为 unchanged 的区域同时是本轮的假设:其他区域会另外给出"除以同一步骤参照区域"后的比较,share_based_speedup 用参照占比推算整体加速。只有让一个步骤内所有区域按同一比例缩放的变化,才会被这样抵消。参照区域本身只做原始比较,不会互相归一化。

合成消融(python tools/measurement_simulation.py,CPU 仿真,非设备实测)

每格 200 次 campaign,数据 seed 20261115;不配对区间这一对照规则用 bootstrap 10000 次。200 次 campaign 的 Monte Carlo 标准误约 0.01–0.03,门槛处的误报以下方精确上界和大样本复核为准。

  • 门槛 1%,A/A: 现行规则判 PASS 的比例为 0.200–0.380;只配对、不算区间为 0.185–0.390;配对精确区间 ≤ 0.010(≥ 6 对)。压低误报的是区间,不是配对。
  • 门槛 0%,A/A(真实中位数恰好在门槛上): 现行规则约 0.45–0.57;配对精确区间 ≤ 0.030,与精确单侧上界(6/7/10/13/20 对分别为 1.56%/0.78%/1.07%/1.12%/2.07%)在 Monte Carlo 误差内一致。Codex 每格 50,000 次复核:6/7/10/13 对的误报为 1.55–1.65%、0.72–0.82%、1.06–1.10%、1.12–1.17%,与精确上界的差距都在 1.56 个标准误以内。
  • 检出力(真实 1.05×,lognormal,20 次重复,门槛 1%): 配对区间 0.880,不配对区间 0.575。配对带来的是检出力。 精确区间比原来的 bootstrap(0.930)略保守,这是换取任何对数下误差都有保证的代价。
  • 5 对及以下不给区间:最极端的一对也有 1/32 ≈ 3.1% 的单侧误差,超过 2.5%。

Checks and dependencies

  • Checks actually run (result and tested scope):
    • ruff check src tests tools:通过。
    • 新 head(基于 main bebd4914)上运行 36 个 harness / campaign / project / dashboard / optimization 相关测试文件(含 tests/test_harness_measurement.py、tests/test_project_explanation.py、tests/test_project_optimization.py、tests/test_optimization_session.py、tests/test_module_boundaries.py、tests/test_target_module_import_gate.py、tests/test_package_root_inventory.py 和 Dashboard 测试):全部通过,无失败。Git 相关测试使用临时 Git 2.49.1(本机 2.43 不支持 --no-lazy-fetch)。
    • 未重跑本地全量 pytest,以 hosted CI 为准。
  • 独立评审: Codex CLI(gpt-6.1-sol,xhigh)本地只读评审,每一轮都用对抗输入去复现问题。
    • 前六轮发现并已修复:门槛 100% 或比值溢出时写出 Infinity;零基线计数在 1 对时就给出确定方向;breakdown 全为零时抛异常;多个参照区域互相归一化;候选在执行期间改写 mechanism.json 或 state.json 中的声明;中断后经 recover 接受伪造的声明(现记录 mechanism_unverified_after_recovery,按未声明处理)。
    • 针对本轮修订的第七轮:精确秩在 n ≤ 300 的 10,800 组有理数检查中无偏差;离散值和并列时区间保守;88 种判定组合和 1000 组默认行为对照通过;未发现依赖已删除字段或旧语义的代码。发现一处 P2:上千对时 2 ** n 转浮点溢出,且 profile 报告异常会改变默认判定。已修复(有理数精确比较;报告出错记为 unavailable)并补回归测试。第八轮复核结论:Ready(1024 对复现通过;注入报告异常后默认判定不变;31,255 组精确秩检查无偏差)。
    • 针对 david6666666 意见的第九轮:原始与归一化方向相反在单参照、多参照下都被拦截,另构造的单参照反例也判为 unverifiable;8,000 组方向组合检查无反向误判;原始样本不足或零基线不会经归一化变成明确方向;assess 与独立 final 都是 unsupported → INCONCLUSIVE、contradicted → FAIL,搜索继续;rebase 后无旧 CLI 路径残留。结论 Ready with fixes,唯一一项是同步本 PR 说明(已同步)。
    • 针对 rebase 冲突和重复次数下限的第十轮:冲突合并保留了两项行为(先验证执行正常结束,再用执行前绑定的声明判定;六种异常执行记录在直接结算和 recover 时都不调用判定器);--require-explanation 下 start 拒绝 3/5 次重复、round 在执行前拒绝 6 次重复下的 4 条预期,返回用法错误,campaign 不变、不耗轮次;k = 1…12 的所需对数与 pairs_needed 完全一致;开关关闭时仍可运行。结论 Ready,无 P0/P1/P2。
    • plan.json 已有 plan_digest 保护。
  • Not run: hosted CI;GPU/NPU 实测;真实 profiler 采集。
  • Dependencies / integration assumptions:
    • measurement.py 只用标准库。
    • 配对依赖 steps() 两侧等长且交替执行;区间的精确性依赖各次重复独立、比值来自连续分布,离散计数出现并列时偏保守。
    • 候选以调用者权限运行,哈希核对是漂移检测,不是沙箱。
    • 修改 harness/*.py 会改变 runtime identity,旧 runtime 启动的会话需要用原来冻结的 runtime 恢复(既有规则)。

评审后的修订

来源 意见 处理
steward 条件 1;amy-why-3459;cursor[bot] rebase 到当前 main 已 rebase 到 0311393c(含 #461 的 CLI 迁移、#474、#323、#477、#466、#483、#475)。optimization.py 的 _settle 与 main 891aadda 冲突:保留 main 新增的"执行必须正常结束"检查,其后用执行前绑定的声明调用判定器;新 head 上重跑测试
steward 条件 2;amy-why-3459;cursor[bot] 两开关优先级;点估计未过而区间过门槛 写入文档和 #454;test_both_switches_apply_the_decision_before_the_explanation(6 例)、test_paired_decision_passes_when_only_the_point_estimate_misses_the_gate
steward 条件 3 默认关闭、只报告 保持;新增 profile 报告出错不改判的回归测试
linyueqian bootstrap 下界是次序统计量,5/7/13 对单侧误差 3.1%/6.25%/4.6% 改为精确次序统计量区间,最少 6 对;记录 order_statistic_rank、achieved_confidence
linyueqian unchanged 不是证据,且没有多重比较校正 Bonferroni(family_size、confidence、pairs_needed);unchanged 只记 not_contradicted / contradicted
linyueqian 跨门槛记 FAIL 与 docs/contributing.md 的 FAIL 定义冲突 改为 INCONCLUSIVE(within_noise);低于门槛仍为 FAIL
linyueqian 区间决定判定时仍写 report_only: true --decision paired 时为 false
david6666666(行内) 原始与归一化方向相反时,归一化方向覆盖原始方向,原始增加 20% 被判为"已解释的下降" 记为 raw_normalized_conflict,该条核对 unverifiable;test_raw_and_normalized_sign_reversal_is_not_read_as_either_direction 复现了他描述的 10 对情形(两个参照各自跨过 1,但总和每对都上升),修复前判 explained,修复后为 unverifiable
david6666666(#454) unsupported 是证据不足,应与 contradicted 区分 --require-explanation 下 unsupported 改为 INCONCLUSIVE,只有 contradicted 记 FAIL;候选都不被采纳
linyueqian(复核) --require-explanation 在少于 6 次重复时仍可启动,任何预期都拿不到区间,没有候选能被采纳;单轮声明所需对数超过冻结重复次数时也注定 unverifiable 开关打开时 campaign start 拒绝少于 6 次重复;round --mechanism-file 在执行前拒绝所需对数(与 profile_evidence 的 pairs_needed 同一公式)超过冻结重复次数的声明,不记录轮次、不耗预算;开关关闭时声明仍只报告
david6666666(行内,project.py) --decision paired 下判定看配对区间,选优仍比较未配对中位数:9 对、1% 门槛时区间 [1.111, 1.111] 判 PASS,但中位数 6 vs 9 使基线仍被选中,final 没有复测该候选 选 A:配对证据同时决定采纳。启动时冻结 selection = paired_median_speedup(maximize,基线 1.0),score/speedup/improvement_pct 取逐对加速比中位数,未配对值留在 point_estimate;每轮记录 selection.outcome 说明是否被选中及原因;默认 point 与题包会话的选优不变。test_paired_decision_also_selects_the_candidate_that_final_retests 端到端复现他的 9 对情形(采纳、final 复测、optimization_speedup 1.111)

Remaining work and Issue closure

Copy link
Copy Markdown
Collaborator

我对照当前 main cd8fcb452bd2c7969a7a6e2544aa1cb5f8ad5332 和本 PR 的 head 1b2b4593cb72ef7b956f82f000313a9910cffbd7 做了完整 review(分支点是 a92c77ccaed68426c83154cff7619745f34d0058,其上 5 个提交:9e01b92bf08e505c259b948633d976019936dcac … 1b2b4593cb72ef7b956f82f000313a9910cffbd7)。#454 仍是 Proposed,这条评论只作设计/实现对照,不是 Approve。

先说已经对上我在 #454 里提的几条:explained / unsupported / contradicted / unverifiable 的机械定义在 explain() 和文档里写清楚了(全部确认才算 explained;任一条方向被打脸即为 contradicted;少于 5 对或零基线计数为 unverifiable);默认关闭时,test_assessment_reports_paired_statistics_without_changing_the_verdict 和 test_explanation_is_report_only_without_the_switch 覆盖了原判定不变;机制在执行前绑定,并覆盖了改写 state.json / mechanism.json,以及 recover 后不信任声明;tools/measurement_simulation.py 已附上,设定(data seed 20261115、门槛 1%、200 次 campaign、bootstrap 10000、seed 0)可以复核。steps() 已按重复交替 AB/BA,completed 的 assess() 两侧样本等长;某一侧测量失败会按既有 incomplete_execution 整轮失败,--decision paired 在统计 unavailable 时是 INCONCLUSIVE,不会退回未配对中位数。

还需要处理的点:

  1. Governing issue 尚未接受。 rfc: 可解释的改进——用 profile 证据和配对统计判定 PASS #454 仍是 Proposed,goal-acceptance owner 为 TBD,三个 owner 决策还没有结论。在 issue 被接受之前,这个草稿适合用来对照设计,但我不会 Approve。另外,rfc: 可解释的改进——用 profile 证据和配对统计判定 PASS #454 正文里还没有补上本 PR 的链接,请补上 https://github.com/InferMatrixLab/InferRSI/pull/457。

  2. 请 rebase 到当前 main。 分支点是 a92c77ccaed68426c83154cff7619745f34d0058,之后 main 已合入 Package shared technical contracts and retire root modules #453(cd8fcb452bd2c7969a7a6e2544aa1cb5f8ad5332)。Package shared technical contracts and retire root modules #453 改了 src/inferrsi/arguments.py 的 RunSpec 导入路径(本 PR 也改了这个文件,hunk 看起来不重叠,但仍需要先 rebase)。正文里本地全量 pytest 的对比基线也是 a92c77cc;rebase 后请按新 head 再跑一遍定向测试,并视情况更新声明里的 head_sha。

  3. 两个开关同时打开时的优先级,以及点估计与区间不一致时的判定,请写进文档并补测试。 这是我在 rfc: 可解释的改进——用 profile 证据和配对统计判定 PASS #454 评论里第 3 点仍未钉死的部分。assess()(src/inferrsi/harness/project.py 第 188–214 行)先按 --decision 得到 verdict,再仅在 verdict == "PASS" 时应用 --require-explanation。因此两者同时打开时,若配对结果已经是 FAIL / within_noise,即使 explanation 是 unverifiable,最终仍是 FAIL,不会变成 INCONCLUSIVE。请在 rfc: 可解释的改进——用 profile 证据和配对统计判定 PASS #454 和 docs/optimization-workflow-unification.md 里写明这条优先级,并加一个同时打开两个开关的回归测试。另外,--decision paired 只看区间(above_gate 即为 PASS,第 192–193 行),所以“点估计未过门槛、但配对区间整体越过门槛”时会判 PASS;这一点目前只是从代码读出来的,请在文档里写一句,并最好补一个 agrees_with_point_gate is False 且区间为 above_gate 的用例。

我没有重新跑 tools/measurement_simulation.py 的 200×10000 仿真,也没有跑 hosted CI。硬件 A/A + 已知改进对照仍是 #454 留给 owner 的决策,不作为这条评论的合入前提。默认关闭、机制绑定和合成测试设计看起来是对口的。

我不会在这个 head 上 Approve,直到 #454 被接受,并且上面的 rebase / 文档与测试缺口处理完毕。

@Dong1017 Dong1017 added area:harness Harness module: profile, A/B loop, quality gates, evidence reports kind:task Bounded delivery task with a delegated slice labels Nov 15, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR review verdict — DO NOT PASS
Reviewed base cd8fcb452bd2c7969a7a6e2544aa1cb5f8ad5332 and head 1b2b4593cb72ef7b956f82f000313a9910cffbd7 (draft).

Acting reviewer: cursor[bot] via standing InferRSI steward delegation. Binding formal review to current head.

Blockers before Approve / ready-for-review:

  1. Governing issue not accepted. #454 is still Proposed with goal-acceptance owner TBD. Steward decision will be posted on #454; do not treat this draft as authorized delivery until that record exists.
  2. Rebase onto current main. Head diverges: ahead 5 / behind 1 vs cd8fcb45 (#453), which also touched src/inferrsi/arguments.py. Rebase and re-run the focused harness tests on the new head; refresh declaration head_sha if needed.
  3. Dual-switch priority and point/interval disagreement are under-specified in docs/tests. In assess() (project.py), --decision runs first; --require-explanation only rewrites a PASS. So with both switches on, within_noise FAIL stays FAIL even if explanation is unverifiable. Also, paired mode keys only on gate_relation (above_gate → PASS), so a point estimate below the gate can still PASS when the interval clears the gate. Document both behaviors in #454 and docs/optimization-workflow-unification.md, and add regression tests (both switches on; agrees_with_point_gate is False with above_gate).

Checked / helpful but not enough to Approve: mechanical explanation.status definitions look consistent with the RFC; defaults-off report-only path is covered; mechanism binding / tamper / recover handling is present; tools/measurement_simulation.py exists with stated seeds. I did not re-run the 200×10000 simulation or GPU/NPU hardware checks. Hosted CI on this draft is limited (declarations green); full ready-lane evidence is still missing.

Hardware A/A + known-gain confirmation is not a merge prerequisite while both switches default off (steward decision on #454). Re-request review after acceptance + rebase + the doc/test gaps above.

Copy link
Copy Markdown
Collaborator

回看了 Dong1017 在 #454 的 steward 决定:三个 owner 决策已经给出(可选能力、默认关闭、按一个 PR 评审),goal-acceptance owner 记为 Dong1017,硬件 A/A 也不再作为合入前提。这一条外部阻塞已经解除。

当前 head 仍是 1b2b4593cb72ef7b956f82f000313a9910cffbd7,这次没有新的提交,所以我上次列出的其余阻塞还在:

  1. 仍需 rebase 到当前 main。 分支点还是 a92c77ccaed68426c83154cff7619745f34d0058。在 Package shared technical contracts and retire root modules #453(cd8fcb452bd2c7969a7a6e2544aa1cb5f8ad5332)之后,main 又合入了 refactor(C2): relocate resource facade and split evidence layout #458(5db1bc94042f386f0b4b35c3e6fe91598fef118f)。请变基到这个新的 main,按新 head 再跑定向 harness 测试,并视情况更新声明里的 head_sha。
  2. 两个开关的优先级、以及点估计与区间不一致时的判定,仍未写入文档并补测试。 这次 head 没有改动,缺口与我在 1b2b4593cb72ef7b956f82f000313a9910cffbd7 上的上次评论相同。

因此我不会在这个 head 上 Approve。cursor[bot] 在同一 head 上也给出了 CHANGES_REQUESTED / DO NOT PASS,结论一致。请在处理完上述两点后(通常会形成新的 head)再提请复核。

@linyueqian

Copy link
Copy Markdown
Contributor

Notes on --decision paired and --require-explanation at 1b2b4593.

The module docstring already says the interval's coverage is not calibrated. Near the 5-pair floor it is off by enough to change what a paired PASS means. At 5, 7 and 13 pairs, bootstrap_median_interval (src/inferrsi/harness/measurement.py L73-L80) returns an order statistic as the lower bound: the smallest pair ratio at 5 pairs, the 2nd smallest at 7 and the 4th smallest at 13. For independent continuous pair ratios whose median equals the gate, PASS then comes with probability P(Bin(n, 1/2) <= j-1): 3.1% at 5 pairs, 6.25% at 7 and 4.6% at 13. The one-sided nominal rate is 2.5%. A 0% gate is a speedup of 1 in both directions, so these are also the A/A rates when the median pair ratio is 1. Your own rates() at 7 repetitions and a 0% gate gives A/A PASS 0.075 (SE 0.008, exact 0.0625) and coverage 0.857 (exact 0.875). The quoted 0.020 holds because the 1% gate sits above no change, and only 5, 10 and 20 pairs ran. A nonrandomized exact sign test keeps the rate at or below 2.5%, and it can first reject at 6 pairs (2^-5 = 3.1%, 2^-6 = 1.6%).

Three smaller points:

  1. A region declared unchanged passes when its interval includes 1 (measurement.py L299-L300). That is not evidence of no change, and there is no multiplicity control. With k such regions and independent noise, a false contradicted comes at 1-(1-q)^k, where q is 6.25% at 5 pairs and 12.5% at 7. That is about 49% for 5 regions at 7 pairs. When the paired decision gives PASS, --require-explanation then turns it into FAIL.
  2. This is separate from the switch-priority question already raised. A straddling interval gives FAIL within_noise (project.py L192-L193), but it does not show that the threshold is missed. docs/contributing.md L138 defines FAIL as evidence that a gate was not met. If FAIL here means "not shown to pass", please say so. Otherwise INCONCLUSIVE would match evidence/qwen38-h20-fp8-single-request-vllm032/validator/statistics-8k-v1/paired_statistics_8k_v1.py L147, which gives INCONCLUSIVE when the point passes and the bound crosses.
  3. Paired records still write report_only: true (measurement.py L134) when the interval decides the verdict.

Both switches are off by default, so this is a good time to fix these.

BeatSeat added a commit to BeatSeat/InferRSI that referenced this pull request Nov 16, 2026
…ithin noise

The percentile bootstrap lower bound for a median of n pairs is an order
statistic whose one-sided error exceeds 2.5% at small n (3.1% at 5 pairs,
6.25% at 7). Use the exact distribution-free order-statistic interval (the
inverted sign test) instead: the rank keeps each tail at or below 2.5%, so the
minimum is six pairs and A/A at the gate passes at most 2.5% of the time.

Declared expectations are judged as a family: region and counter intervals use
the Bonferroni confidence 1 - 5%/k, and too few pairs for it is unverifiable.
A declared-unchanged item is reported as not contradicted, never confirmed.

A paired interval that straddles the gate shows only that passing was not
demonstrated, which is not evidence that the gate was missed, so the round is
INCONCLUSIVE (within_noise) and still does not block the search. Records say
report_only: false when the interval decides the verdict.

Reported by linyueqian on InferMatrixLab#457.

Agent-assisted-by: Claude Code / claude-opus-5-5 (author)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
@BeatSeat
BeatSeat force-pushed the feat/paired-measurement-statistics branch from 1b2b459 to ac0510b Compare November 16, 2026 09:17
@BeatSeat

Copy link
Copy Markdown
Author

已在新 head ac0510b2 上处理三位评审的意见(基于当前 main 016c7c7d)。

amy-why-3459 / cursor[bot] / steward 条件

  1. Rebase:已 rebase 到 016c7c7d(Package shared technical contracts and retire root modules #453、refactor(C2): relocate resource facade and split evidence layout #458、refactor(R1): retire eleven roots and bind Apps inspection #459、Extend VLA observation-to-action benchmark guidance #342、Wire the Harness loop evaluation port through Apps #462、knowledge: reject foreign records in a source-snapshot pack #463 之后),没有冲突;按新 head 重跑了定向 harness 测试和相关 campaign/dashboard 测试,见 PR 正文。
  2. 两开关优先级:--decision 先给出性能结论,--require-explanation 只把 PASS 降为 FAIL 或 INCONCLUSIVE,不改写非 PASS 的结论。已写入 docs/optimization-workflow-unification.md 和 rfc: 可解释的改进——用 profile 证据和配对统计判定 PASS #454。test_both_switches_apply_the_decision_before_the_explanation 覆盖 6 种组合(区间跨门槛 + unverifiable / 未声明、区间低于门槛 + 未支持、配对 PASS + 未声明 / 矛盾 / 已解释)。
  3. 点估计未过、区间过门槛:--decision paired 只看区间,此时判 PASS 并记录 agrees_with_point_gate: false。test_paired_decision_passes_when_only_the_point_estimate_misses_the_gate 用 10 对数据构造点估计 +0.5%(未过 1% 门槛)、区间 [1.07, 1.07] 的情形。
  4. 默认关闭时只报告:原有回归测试保留,report_only: true;新增 test_failing_profile_report_does_not_change_the_default_verdict,profile 报告出错时记为 unavailable,不改判定。

linyueqian(谢谢,复算全部成立)

  1. 区间的单侧误差:改用精确的次序统计量区间(符号检验反演),秩 j 取使 P(Bin(n, 1/2) ≤ j−1) ≤ 2.5% 的最大值,并记录 order_statistic_rank 和 achieved_confidence。MIN_PAIRS 由 5 改为 6,--decision paired 至少要 6 次重复。门槛为 0% 时的 A/A 仿真结果见 PR 正文,每个对数都不超过对应的精确上界;test_interval_rank_is_the_largest_exact_rank_within_two_and_a_half_percent 对 n = 1…40 逐一核对秩的最大性。
  2. "不变"与多重比较:声明 k 条预期时,所有区域和计数使用 Bonferroni 置信度 1 − 5%/k(记录 family_size、confidence、pairs_needed),对数不足时为 unverifiable。每条核对记录 outcome;unchanged 只会是 not_contradicted 或 contradicted,不会被"确认不变"。explain() 拒绝按更少条数校正的证据。
  3. 区间跨过门槛:按 docs/contributing.md 的定义改为 INCONCLUSIVE(within_noise),区间整体低于门槛仍为 FAIL,与 qwen38 验证器的处理一致。这类轮次不被采纳,也不阻塞搜索。
  4. report_only:--decision paired 时为 false。

另外 Codex CLI(gpt-6.1-sol,xhigh)对新 head 做了一轮只读对抗评审,结论和处理见 PR 正文。请在新 head 上复核。

amy-why-3459
amy-why-3459 previously approved these changes Nov 16, 2026

@amy-why-3459 amy-why-3459 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 016c7c7df023944f0f46d8e777d6d31f1df1764c and head ac0510b21b887e34b7357b13dc3d2eb7c9e86c5f(draft)。

上次在 1b2b4593cb72ef7b956f82f000313a9910cffbd7 上的阻塞已处理:

  1. #454 已附条件接受(steward 评论 6897439035)。
  2. 已 rebase 到当前 main 016c7c7d(ahead 8 / behind 0)。
  3. 两开关优先级与「点估计未过、区间过门槛」已写入 docs/optimization-workflow-unification.md 和 #454,并有 test_both_switches_apply_the_decision_before_the_explanation、test_paired_decision_passes_when_only_the_point_estimate_misses_the_gate 覆盖。

本轮还核对了 linyueqian 在旧 head 上的意见及后续修订:精确次序统计量区间、MIN_PAIRS=6、Bonferroni 家族校正、跨门槛记 INCONCLUSIVE(within_noise)、--decision paired 时 report_only: false,以及 profile 报告失败不改变默认判定。我用 n=1…40 的二项尾计算复核了 interval_rank,与文档中的对数要求(k=1 为 6,k=2–3 为 7,k=4–6 为 8,k=7–12 为 9)一致。declarations 已通过。我没有重跑 tools/measurement_simulation.py 的仿真,也没有跑 hosted 全量 pytest。

@david6666666 david6666666 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One issue with how normalized evidence affects the explanation gate.

[v / k for v, k in zip(c, clock[1])],
confidence=confidence)
entry.update(normalized=normalized, observed=normalized["observed"],
basis="normalized_to_reference_regions")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could a sign reversal between raw and normalized evidence stay inconclusive unless the mechanism explicitly predicts a relative effect? This overwrites the raw direction. In a 10-pair check, the raw region interval shows a 20% increase, the normalized interval shows a decrease, and the reference interval merely crosses 1. The result is explained for an absolute decrease. Please preserve that disagreement in the explanation gate and add a regression test.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, agreed and fixed in 6a3c1236. When a region's raw and reference-normalized intervals point in opposite definite directions, the evidence now records observed: "raw_normalized_conflict" and keeps both raw and normalized. That check's outcome is unverifiable, so explanation.status is unverifiable (INCONCLUSIVE under --require-explanation). The normalized sign no longer wins. The mechanism format has no way to declare a relative-only effect, so a reversal is never read as either direction.

A raw no-clear-change with a definite normalized direction is still resolved by normalization; that's the drift it is meant to cancel. A definite raw direction with no clear normalized change stays not_shown.

Regression: test_raw_and_normalized_sign_reversal_is_not_read_as_either_direction reproduces your 10-pair case. The raw target is +20%; two references each cross 1 in different pairs, so neither is contradicted, yet their sum rises in every pair. The normalized result is a decrease. Before the fix this was explained; now it is unverifiable. test_normalization_still_resolves_a_raw_change_hidden_by_drift keeps the intended normalization path. Codex also built a single-reference variant and checked 8,000 direction combinations without finding another reversal.

cursor[bot]
cursor Bot previously approved these changes Nov 16, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 016c7c7 and head ac0510b (draft).

Re-review after author push from prior CHANGES_REQUESTED on 1b2b459. Prior blockers cleared on this head:

  1. Governing #454 acceptance already recorded.
  2. Rebased onto 016c7c7 with focused harness tests reported in the PR body.
  3. Dual-switch priority and point/interval disagreement documented in docs/optimization-workflow-unification.md and covered by test_both_switches_apply_the_decision_before_the_explanation plus test_paired_decision_passes_when_only_the_point_estimate_misses_the_gate (agrees_with_point_gate is False). Defaults-off report-only path retained.

Merge process (not content blockers on this draft head): still draft; head is behind current main again after later merges — rebase before marking ready; run ready-lane CI on the rebased tip. Hardware A/A remains non-blocking while switches default off.

Acting steward re-review while hosted Dong-bot is on usage_limit; formal Approve as cursor[bot].

BeatSeat added a commit to BeatSeat/InferRSI that referenced this pull request Nov 16, 2026
…ithin noise

The percentile bootstrap lower bound for a median of n pairs is an order
statistic whose one-sided error exceeds 2.5% at small n (3.1% at 5 pairs,
6.25% at 7). Use the exact distribution-free order-statistic interval (the
inverted sign test) instead: the rank keeps each tail at or below 2.5%, so the
minimum is six pairs and A/A at the gate passes at most 2.5% of the time.

Declared expectations are judged as a family: region and counter intervals use
the Bonferroni confidence 1 - 5%/k, and too few pairs for it is unverifiable.
A declared-unchanged item is reported as not contradicted, never confirmed.

A paired interval that straddles the gate shows only that passing was not
demonstrated, which is not evidence that the gate was missed, so the round is
INCONCLUSIVE (within_noise) and still does not block the search. Records say
report_only: false when the interval decides the verdict.

Reported by linyueqian on InferMatrixLab#457.

Agent-assisted-by: Claude Code / claude-opus-5-5 (author)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
BeatSeat added a commit to BeatSeat/InferRSI that referenced this pull request Nov 16, 2026
When a region's raw and reference-normalized intervals point in opposite
definite directions, an absolute increase can read as a relative decrease
(two references can each cross 1 while their sum rises in every pair). No
declaration predicts a relative effect, so record raw_normalized_conflict and
leave that check unverifiable instead of letting the normalized sign win.

Under --require-explanation, only contradicted evidence fails. An unsupported
prediction is missing evidence, like a paired interval within noise, so it is
INCONCLUSIVE; the candidate is still not adopted and the search continues.

Reported by david6666666 on InferMatrixLab#454 and InferMatrixLab#457.

Agent-assisted-by: Claude Code / claude-opus-5-5 (author)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
@BeatSeat
BeatSeat dismissed stale reviews from cursor[bot] and amy-why-3459 via 6a3c123 November 16, 2026 13:46
@BeatSeat
BeatSeat force-pushed the feat/paired-measurement-statistics branch from ac0510b to 6a3c123 Compare November 16, 2026 13:46
@BeatSeat

Copy link
Copy Markdown
Author

New head 6a3c1236, rebased onto current main 6c7e2379 (including #461's CLI move; the --decision, --require-explanation and --mechanism-file options now live in src/inferrsi/apps/cli/arguments.py), as cursor[bot]'s merge-process note asked.

It also addresses david6666666's two points:

The 34 related test files and ruff pass on this head. A Codex review of the change found no code issues; its only item was syncing the PR body, which is now done. Please re-review on 6a3c1236.

amy-why-3459
amy-why-3459 previously approved these changes Nov 16, 2026

@amy-why-3459 amy-why-3459 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 6c7e237935a22cc5e90e954d4f397376e753e7d8 and head 6a3c123691b0ba6937084837c62a888b9735f59b(draft)。

对照上次已通过的 ac0510b21b887e34b7357b13dc3d2eb7c9e86c5f,本轮增量主要是 6a3c123691b0ba6937084837c62a888b9735f59b,并已 rebase 到 main 6c7e2379(含 #461,--decision / --require-explanation / --mechanism-file 现位于 src/inferrsi/apps/cli/arguments.py)。

已核对 david6666666 的两点:

  1. --require-explanation 下只有 contradicted 记 FAIL,unsupported / unverifiable / not_declared 记 INCONCLUSIVE(src/inferrsi/harness/project.py 与 tests/test_project_explanation.py)。
  2. raw/normalized 方向冲突记为 raw_normalized_conflict,_check_outcome 将其判为 unverifiable;回归测试 test_raw_and_normalized_sign_reversal_is_not_read_as_either_direction 覆盖了所述 10 对情形,test_normalization_still_resolves_a_raw_change_hidden_by_drift 保留了归一化抵消漂移的路径。文档与 #454 已同步。

合并流程(非本 head 内容阻塞):仍为 draft;相对当前 main 2cbfab66b00d75e69be583ebfeabee2b8f83c4ea(#470/#472)落后 2 个提交,但无文件重叠,GitHub 显示可合并无冲突。标记 ready 前请再 rebase。我没有重跑作者声称通过的相关测试和 ruff,也没有跑 hosted 全量 pytest。

cursor[bot]
cursor Bot previously approved these changes Nov 16, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 6c7e237935a22cc5e90e954d4f397376e753e7d8 and head 6a3c123691b0ba6937084837c62a888b9735f59b (draft).
Re-review after author push from prior PASS on ac0510b21b88 (Approves dismissed by rebase). Incremental head addresses david6666666: raw/normalized definite sign reversal records raw_normalized_conflict → check unverifiable; under --require-explanation only contradicted is FAIL and unsupported/unverifiable/not_declared stay INCONCLUSIVE. Prior blockers remain cleared (#454 accepted; dual-switch priority + point/interval disagreement documented/tested). Still draft; behind current main by 2 non-overlapping commits — rebase before ready. Formal Approve as cursor[bot].

@linyueqian

Copy link
Copy Markdown
Contributor

Re-checked on 6a3c1236. All four points hold.

  1. Rank: interval_rank (measurement.py L29-L41) returns the largest rank with P(Bin(n, 1/2) <= j-1) <= 2.5%. I checked every n from 6 to 60 with exact fractions. For independent continuous pairs, the PASS rate at the gate stays at or below 2.5% (the highest is 2.45% at 17 pairs), and 5 or fewer pairs give no interval.
  2. Multiplicity: with 1 predicted decrease plus 1, 2 or 4 unchanged regions with independent noise and no shared drift, a false contradicted came at 0.3% to 2.9% (20,000 runs each, at pairs_needed and at 10 pairs). That is under the 5% family bound.
  3. A straddling interval is INCONCLUSIVE within_noise, and an interval below the gate is FAIL (project.py L193-L197).
  4. report_only is not decides (measurement.py L158), and assess sets decides for --decision paired (project.py L184).

The four changed test files pass on this head (160 passed).

One new gap. validate() checks the repetition floor only for --decision paired (project.py L42-L43). With the default point decision, a campaign with --require-explanation and 3 or 5 repetitions starts. No expectation can then get an interval (insufficient_pairs), so the explanation is unverifiable (or not_declared without a mechanism). Every PASS becomes INCONCLUSIVE, and no candidate can be adopted. In my check, 5 pairs where kernel_launches drops from 20 to 10 in every pair give unverifiable. The same happens per round: the 3-item example in the docs needs 7 pairs, so at 6 repetitions it is unverifiable and at 7 it is explained. When --require-explanation is on, could start reject fewer than MIN_PAIRS repetitions? And could round --mechanism-file (optimization_session.py L220-L226) reject a declaration whose pairs_needed is above the frozen repetitions? The plan with those repetitions is already loaded at L203, so the round would not spend its budget first. Without the switch, a low-repetition mechanism stays a report and can still run.

BeatSeat added a commit to BeatSeat/InferRSI that referenced this pull request Nov 17, 2026
…ithin noise

The percentile bootstrap lower bound for a median of n pairs is an order
statistic whose one-sided error exceeds 2.5% at small n (3.1% at 5 pairs,
6.25% at 7). Use the exact distribution-free order-statistic interval (the
inverted sign test) instead: the rank keeps each tail at or below 2.5%, so the
minimum is six pairs and A/A at the gate passes at most 2.5% of the time.

Declared expectations are judged as a family: region and counter intervals use
the Bonferroni confidence 1 - 5%/k, and too few pairs for it is unverifiable.
A declared-unchanged item is reported as not contradicted, never confirmed.

A paired interval that straddles the gate shows only that passing was not
demonstrated, which is not evidence that the gate was missed, so the round is
INCONCLUSIVE (within_noise) and still does not block the search. Records say
report_only: false when the interval decides the verdict.

Reported by linyueqian on InferMatrixLab#457.

Agent-assisted-by: Claude Code / claude-opus-5-5 (author)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
BeatSeat added a commit to BeatSeat/InferRSI that referenced this pull request Nov 17, 2026
When a region's raw and reference-normalized intervals point in opposite
definite directions, an absolute increase can read as a relative decrease
(two references can each cross 1 while their sum rises in every pair). No
declaration predicts a relative effect, so record raw_normalized_conflict and
leave that check unverifiable instead of letting the normalized sign win.

Under --require-explanation, only contradicted evidence fails. An unsupported
prediction is missing evidence, like a paired interval within noise, so it is
INCONCLUSIVE; the candidate is still not adopted and the search continues.

Reported by david6666666 on InferMatrixLab#454 and InferMatrixLab#457.

Agent-assisted-by: Claude Code / claude-opus-5-5 (author)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
BeatSeat added a commit to BeatSeat/InferRSI that referenced this pull request Nov 17, 2026
With --require-explanation, fewer than six repetitions give no interval for
any expectation, so every round would be unverifiable and nothing could be
adopted; campaign start now rejects that. A round whose declaration needs more
pairs than the frozen repetitions (Bonferroni over its expectations) is
rejected before execution, so it spends no budget. Without the switch a
declaration remains a report and is not restricted.

Reported by linyueqian on InferMatrixLab#457.

Agent-assisted-by: Claude Code / claude-opus-5-5 (author)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
@BeatSeat
BeatSeat dismissed stale reviews from cursor[bot] and amy-why-3459 via 60ba2d0 November 17, 2026 01:47
@BeatSeat
BeatSeat force-pushed the feat/paired-measurement-statistics branch from 6a3c123 to 60ba2d0 Compare November 17, 2026 01:47
@BeatSeat

Copy link
Copy Markdown
Author

Thanks for the re-check, and agreed on the new gap. Fixed in 60ba2d0d, on main bebd4914:

  • Start: with --require-explanation, campaign start rejects fewer than MIN_PAIRS (6) repetitions, whatever --decision is. Without the switch, low repetition counts still start (report only).
  • Round: with the switch on, round --mechanism-file computes min_pairs(1 - 5%/k) for the declaration's k expectations. That is the same formula as profile_evidence's pairs_needed. If it exceeds the frozen repetitions, the round is rejected before execution: no round is recorded, no budget is spent, and the campaign stays active. Your docs example has 3 items and needs 7 pairs, so it is rejected at 6 repetitions; the 4-item FUSION example needs 8 pairs and is rejected at 6. Without the switch, the same declaration runs and is reported as unverifiable.
  • Tests: test_require_explanation_rejects_too_few_repetitions_at_start, test_uncheckable_declaration_is_rejected_before_it_spends_a_round and test_uncheckable_declaration_stays_a_report_without_the_switch. The docs are updated.

The rebase onto bebd4914 had one conflict, in _settle. I kept main's new check that execution finished normally (891aadda), then the assess call with the pre-execution declaration. The 36 related test files and ruff pass. A Codex review of this head found no issues: the merge keeps both behaviours, and pairs_needed matches the round check for k = 1…12.

@david6666666 david6666666 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One case where the paired verdict and campaign selection disagree.

Comment thread src/inferrsi/harness/project.py Outdated
"completed", "fixed_checks_satisfied" if accepted else "performance_gate_not_met",
verdict="PASS" if accepted else "FAIL", score=after, baseline_score=before,
"completed", reason,
verdict=verdict, score=after, baseline_score=before,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In --decision paired mode, this still hands the selector the unpaired median. In a nine-pair minimize fixture at a 1% gate, the paired interval was about [1.111, 1.111] (PASS, eligible=true), but medians 6 vs 9 left the baseline selected, so final never validated the passing candidate. Can we define whether paired evidence controls adoption as well as the verdict and add a campaign-level test for this disagreement? If point-score ranking is intentional, please document the veto and report why the passing round was not selected.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, confirmed and fixed in 507ce154 with your option A (the branch is also rebased onto current main 0311393c). I reproduced it exactly: baseline [1,2,3,4,6,10,10,10,10], candidate [0.9,1.8,2.7,3.6,100,9,9,9,9]. That gives the interval [1.111, 1.111] and a PASS, with medians 6 vs 9, and _settle kept the baseline. Paired evidence now controls adoption as well as the verdict.

  • Frozen selection. Under --decision paired, campaign start freezes selection = {"metric": "paired_median_speedup", "direction": "maximize", "initial_score": 1.0} in the digest-bound plan, so the basis can't change mid-search. Each round's score (and speedup) is its median paired speedup against the frozen baseline, and the baseline counts as 1.0. A candidate is adopted when it is PASS and that speedup beats the current best. Every round re-measures the frozen baseline in alternating order, so speedups from different rounds are all relative to the same baseline and comparable.

  • No controller special case. The generic _settle comparison is unchanged. Paired mode reuses the same maximize / initial_score: 1.0 selection shape the benchmark sessions already use, and default point mode and benchmark sessions select exactly as before.

  • Headline numbers follow the decision. improvement_pct is derived from the paired speedup. The unpaired values remain in point_estimate, and the raw medians remain in metrics, so summary.optimization_speedup after final is the paired estimate. Before, a selected paired PASS would have been reported as 0.667×. If the paired statistics are unavailable (for example, a paired ratio underflows even though every sample is finite and positive), the round is INCONCLUSIVE. Its score, speedup and improvement_pct are left empty rather than filled with point values relabelled as paired, and the round is not adopted. The controller accepts a completed round without a score only when it is explicitly eligible: false, so the search continues instead of blocking. test_unavailable_paired_statistics_neither_block_nor_borrow_a_point_score covers this for a round and for final.

  • Why a round was not selected. For the reporting half of your question: every round record now carries selection: {outcome, metric, score, best_score_before}. outcome is one of adopted, not_eligible, not_better_than_best or baseline_identity. A PASS that is outranked by an earlier, faster candidate is recorded as not_better_than_best rather than left implicit. This is additive and applies in both modes.

  • Extreme values settle. In paired mode, a point value that can't be represented as a finite float (an overflowing median, improvement or speedup) is recorded as null instead of failing the round, because the decision doesn't read it. If the paired median itself is unavailable, the round is INCONCLUSIVE and recorded, as above. Default point mode still treats a non-finite point estimate as an infrastructure failure.

Campaign-level regression: test_paired_decision_also_selects_the_candidate_that_final_retests runs your nine-pair minimize fixture at the 1% gate through real campaign start/round/final:

  • Round 1 shows medians 6/9, a point estimate of −50% and paired PASS at 1.111. It is adopted, with best_score_before 1.0.
  • Round 2 is a 1.05× PASS. It is recorded as not_better_than_best.
  • final retests the round-1 candidate (PASS), and optimization_speedup is 1.111.
  • The managed dashboard still labels the medians with the measured metric (metric, ms, minimize) and shows the round as adopted at +10%.

Documented in docs/optimization-workflow-unification.md and the --decision help.

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR review verdict — DO NOT PASS

独立检查 base bebd491 / head 60ba2d0。本次以 COMMENT 提交可复现问题,不作批准。

[P2] paired 判定与 controller 选优仍使用不同的改善标准
位置:src/inferrsi/harness/project.py:225-228;调用方 src/inferrsi/harness/optimization.py:275-285。
paired 模式可以返回 PASS / eligible=true,但 score 仍是候选的独立中位数。controller 随后又要求该 score 优于 baseline/best 的独立中位数,因此已通过配对门槛的首个候选仍可能被丢弃,final 将重测 baseline 而非该候选。

已用真实 campaign start/round 路径复现:10 次测量、minimize、1% 门槛、decision=paired;baseline 测量值为 11..20,candidate 逐项除以 1.03,仅把第一项改为 200。九对加速比是 1.03,配对区间高于门槛,assessment 为 PASS/eligible=true;但 baseline_score=15.5、score=16.019417475728154、improvement_pct=-3.3510804885687384,state.best.kind 仍为 baseline。现有“不通过点门槛但通过 paired”测试只覆盖独立中位数仍有正改善的情况,未覆盖这里的符号反转。

请使 paired 模式的候选选择与其声明的配对判定语义一致,至少补充上述首个合格候选及 final 选择的集成测试;若保留独立中位数作为第二层筛选是有意的,应明确记录这一额外筛选及 rejection 原因,而不能把该模式表述为只看配对区间。

实际验证:本轮重新执行 tests/test_harness_measurement.py、tests/test_project_explanation.py、tests/test_project_optimization.py、tests/test_optimization_session.py,共169项通过;ruff check src tests tools 与 git diff --check 通过。额外独立回归用例确认上述不采纳行为(期望选中候选的断言失败)。合成消融工具以 campaigns=20 / resamples=100 跑完70组,仅作为软件烟测,不作统计精度或硬件性能证明。
未执行:完整仓库测试、完整默认规模消融、GPU/NPU或服务实测、精确head的远端CI验证。需求核验限制:已获得隔离的功能范围,但未取得可独立溯源的原始统计方法、状态映射、重复次数与开关优先级细则,因此不声称完整验收通过。

BeatSeat and others added 11 commits November 17, 2026 22:34
Pair the k-th baseline and candidate samples that steps() already
alternates, and report the median paired speedup with a seeded percentile
bootstrap interval and its relation to the frozen gate. The field is
report-only: verdict, eligibility and selection are unchanged, and any value
that cannot be computed is reported as unavailable so records stay strict
JSON. Fewer than five pairs report no interval because simulated coverage at
three pairs is about 75%. tools/measurement_simulation.py reproduces the
synthetic ablation.

Agent-assisted-by: Claude Code / claude-opus-5-5 (author)
Agent-assisted-by: Codex CLI / gpt-6.1-sol [effort=high] (review)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
…ches

Measurement scripts may add per-step region breakdowns and work counters.
A round may declare its mechanism and expected region/counter directions.
Regions declared unchanged act as an in-step clock, so ratios to them and
their share of the step total cancel device-wide clock or thermal drift;
counters such as launches or DRAM bytes show whether a small gain is
reproducible. Directions come only from paired intervals, with no
repository-wide tolerance.

Two switches are frozen at campaign start and default to the existing
behaviour: --require-explanation adopts only candidates whose mechanism is
confirmed by the data, and --decision paired requires the paired interval
to clear the gate, failing within-noise rounds without blocking the search.

Agent-assisted-by: Claude Code / claude-opus-5-5 (author)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
… evidence

Bind the round's mechanism into control state before the candidate runs and
assess against that copy, so a candidate cannot rewrite its own declaration.
Require five pairs before any profile direction, treat zero-baseline ratios as
undefined, report unavailable shares instead of raising, keep raw evidence for
every region, and never normalize declared references against each other.

Agent-assisted-by: Claude Code / claude-opus-5-5 (author)
Agent-assisted-by: Codex CLI / gpt-6.1-sol [effort=xhigh] (review)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
The candidate runs with the caller's privileges and can write state.json or
the attempt copy. Pass the declaration held since submission to the assessor,
record its digest, and treat a copy that no longer matches as an integrity
failure.

Agent-assisted-by: Claude Code / claude-opus-5-5 (author)
Agent-assisted-by: Codex CLI / gpt-6.1-sol [effort=xhigh] (review)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
An interrupted attempt is settled by a new process that can only read files
the candidate could write. Settle it without the declaration, mark the record,
and leave require_explanation to keep the round unadopted.

Agent-assisted-by: Claude Code / claude-opus-5-5 (author)
Agent-assisted-by: Codex CLI / gpt-6.1-sol [effort=xhigh] (review)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
With both switches on, --decision sets the performance verdict first and
--require-explanation only downgrades a PASS. The paired decision reads the
interval alone, so a candidate whose point estimate misses the gate but whose
paired interval clears it passes, with agrees_with_point_gate recorded false.
Regression tests pin both behaviors.

Agent-assisted-by: Claude Code / claude-opus-5-5 (author)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
…ithin noise

The percentile bootstrap lower bound for a median of n pairs is an order
statistic whose one-sided error exceeds 2.5% at small n (3.1% at 5 pairs,
6.25% at 7). Use the exact distribution-free order-statistic interval (the
inverted sign test) instead: the rank keeps each tail at or below 2.5%, so the
minimum is six pairs and A/A at the gate passes at most 2.5% of the time.

Declared expectations are judged as a family: region and counter intervals use
the Bonferroni confidence 1 - 5%/k, and too few pairs for it is unverifiable.
A declared-unchanged item is reported as not contradicted, never confirmed.

A paired interval that straddles the gate shows only that passing was not
demonstrated, which is not evidence that the gate was missed, so the round is
INCONCLUSIVE (within_noise) and still does not block the search. Records say
report_only: false when the interval decides the verdict.

Reported by linyueqian on InferMatrixLab#457.

Agent-assisted-by: Claude Code / claude-opus-5-5 (author)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
A float 2 ** pairs overflows near a thousand pairs; compare binomial tails as
exact rationals instead. A failure while building profile evidence is reported
as unavailable rather than raised, so it cannot change the default verdict.

Found by Codex review of the exact-interval change.

Agent-assisted-by: Claude Code / claude-opus-5-5 (author)
Agent-assisted-by: Codex CLI / gpt-6.1-sol [effort=xhigh] (review)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
When a region's raw and reference-normalized intervals point in opposite
definite directions, an absolute increase can read as a relative decrease
(two references can each cross 1 while their sum rises in every pair). No
declaration predicts a relative effect, so record raw_normalized_conflict and
leave that check unverifiable instead of letting the normalized sign win.

Under --require-explanation, only contradicted evidence fails. An unsupported
prediction is missing evidence, like a paired interval within noise, so it is
INCONCLUSIVE; the candidate is still not adopted and the search continues.

Reported by david6666666 on InferMatrixLab#454 and InferMatrixLab#457.

Agent-assisted-by: Claude Code / claude-opus-5-5 (author)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
With --require-explanation, fewer than six repetitions give no interval for
any expectation, so every round would be unverifiable and nothing could be
adopted; campaign start now rejects that. A round whose declaration needs more
pairs than the frozen repetitions (Bonferroni over its expectations) is
rejected before execution, so it spends no budget. Without the switch a
declaration remains a report and is not restricted.

Reported by linyueqian on InferMatrixLab#457.

Agent-assisted-by: Claude Code / claude-opus-5-5 (author)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
Under --decision paired the verdict read the paired interval, but adoption
still compared independent candidate medians, so a paired PASS could be
outranked and never reach final. The frozen selection is now the median
paired speedup against the baseline (1.0), and that same speedup is the
round's score, speedup and improvement. Point values stay in point_estimate.
Each round records why it was or was not selected. The default point mode
and benchmark sessions keep their selection.

Agent-assisted-by: Claude Code / claude-opus-5-5 (author)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
@BeatSeat
BeatSeat force-pushed the feat/paired-measurement-statistics branch from 60ba2d0 to 507ce15 Compare November 17, 2026 15:34

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent incremental review, COMMENT: 60ba2d0 → 507ce15, current base c6678fa. Previous independent result: #457 (review). I did not use other review opinions or author-result narratives.

The previously reported paired-selection mismatch is addressed in the inspected code:

  • project.assess now retains independent-median measurements separately as point_estimate and, for paired decisions, supplies median paired speedup as score with baseline_score=1.0.
  • Project session initialization freezes paired selection as maximize / paired_median_speedup / initial_score=1.0, rather than feeding a raw latency objective to the controller.
  • The controller records selection outcome and its comparison basis. Final execution still takes the selected candidate path; new source-level regression coverage explicitly checks that the paired candidate is selected and independently retested at final. I inspected those tests but did not run them here.

Fresh execution was deliberately bounded to read-only, in-memory Windows/Python probes because the cloud test workspace is offline:

  1. Executed the exact-head paired_statistics implementation and assess function with synthetic step-reader/profile stubs. Reused my previous counterexample: baseline11..20, candidate baseline/1.03 with the first value replaced by200. Result: PASS, eligible=true, score1.03, baseline_score1.0, score_basis paired_median_speedup; the independent point improvement remains -3.3510804885687384%. Thus the negative point estimate is no longer the selection objective.
  2. Executed the exact current controller round-selection block in memory with synthetic state and candidate-identity stub. It adopts that score1.03 candidate against baseline1.0.
    These two probes passed. They are not pytest results, a full campaign start/round/final integration run, or a statistical/hardware validation. No repository or campaign files were written.

Machine evidence: independently inspected declarations jobs134155137709/run44116293063 and134153622318/run44115867662. Both evaluate exact head507ce154bc80e68e0a303e26af8a4db79f4313d4 with policyc6678fa8df0e2ab423587eba06151e8386b5b9ce and report compliant/job_exit0. These are declaration-policy results, not product-test passes. The observed current-head check list contained those declaration checks; I have no current-head full-suite/installed/browser pass to certify.

Remaining verification: rerun the new paired-selection/final integration tests and relevant measurement/project/session regression modules on a complete checkout, then inspect applicable current-head CI. I did not rerun pytest, Ruff, diff-check, complete ablation, installed packages, or real GPU/NPU/service work this round. The original statistical-method/state-mapping/repetition/switch-precedence acceptance provenance remains incomplete; I do not silently treat the new implementation or tests as its accepted specification. Other incoming-main changes were not certified by this focused review.

Conclusion: the old mismatch has direct code and bounded executable evidence of correction, but end-to-end and original-AC gaps remain. COMMENT, not APPROVE; no merge, code edit, experiment restart or deployment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:harness Harness module: profile, A/B loop, quality gates, evidence reports kind:task Bounded delivery task with a delegated slice

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants