Repository navigation
Conversation
|
我对照当前 main 先说已经对上我在 #454 里提的几条: 还需要处理的点:
我没有重新跑 我不会在这个 head 上 Approve,直到 #454 被接受,并且上面的 rebase / 文档与测试缺口处理完毕。 |
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Reviewed base cd8fcb452bd2c7969a7a6e2544aa1cb5f8ad5332 and head 1b2b4593cb72ef7b956f82f000313a9910cffbd7 (draft).
Acting reviewer: cursor[bot] via standing InferRSI steward delegation. Binding formal review to current head.
Blockers before Approve / ready-for-review:
- Governing issue not accepted. #454 is still Proposed with goal-acceptance owner TBD. Steward decision will be posted on #454; do not treat this draft as authorized delivery until that record exists.
- Rebase onto current main. Head diverges: ahead 5 / behind 1 vs
cd8fcb45(#453), which also touchedsrc/inferrsi/arguments.py. Rebase and re-run the focused harness tests on the new head; refresh declarationhead_shaif needed. - Dual-switch priority and point/interval disagreement are under-specified in docs/tests. In
assess()(project.py),--decisionruns first;--require-explanationonly rewrites aPASS. So with both switches on,within_noiseFAIL stays FAIL even if explanation isunverifiable. Also, paired mode keys only ongate_relation(above_gate→ PASS), so a point estimate below the gate can still PASS when the interval clears the gate. Document both behaviors in #454 anddocs/optimization-workflow-unification.md, and add regression tests (both switches on;agrees_with_point_gate is Falsewithabove_gate).
Checked / helpful but not enough to Approve: mechanical explanation.status definitions look consistent with the RFC; defaults-off report-only path is covered; mechanism binding / tamper / recover handling is present; tools/measurement_simulation.py exists with stated seeds. I did not re-run the 200×10000 simulation or GPU/NPU hardware checks. Hosted CI on this draft is limited (declarations green); full ready-lane evidence is still missing.
Hardware A/A + known-gain confirmation is not a merge prerequisite while both switches default off (steward decision on #454). Re-request review after acceptance + rebase + the doc/test gaps above.
|
回看了 Dong1017 在 #454 的 steward 决定:三个 owner 决策已经给出(可选能力、默认关闭、按一个 PR 评审),goal-acceptance owner 记为 Dong1017,硬件 A/A 也不再作为合入前提。这一条外部阻塞已经解除。 当前 head 仍是
因此我不会在这个 head 上 Approve。cursor[bot] 在同一 head 上也给出了 CHANGES_REQUESTED / DO NOT PASS,结论一致。请在处理完上述两点后(通常会形成新的 head)再提请复核。 |
|
Notes on The module docstring already says the interval's coverage is not calibrated. Near the 5-pair floor it is off by enough to change what a paired PASS means. At 5, 7 and 13 pairs, Three smaller points:
Both switches are off by default, so this is a good time to fix these. |
…ithin noise The percentile bootstrap lower bound for a median of n pairs is an order statistic whose one-sided error exceeds 2.5% at small n (3.1% at 5 pairs, 6.25% at 7). Use the exact distribution-free order-statistic interval (the inverted sign test) instead: the rank keeps each tail at or below 2.5%, so the minimum is six pairs and A/A at the gate passes at most 2.5% of the time. Declared expectations are judged as a family: region and counter intervals use the Bonferroni confidence 1 - 5%/k, and too few pairs for it is unverifiable. A declared-unchanged item is reported as not contradicted, never confirmed. A paired interval that straddles the gate shows only that passing was not demonstrated, which is not evidence that the gate was missed, so the round is INCONCLUSIVE (within_noise) and still does not block the search. Records say report_only: false when the interval decides the verdict. Reported by linyueqian on InferMatrixLab#457. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
1b2b459 to
ac0510b
Compare
|
已在新 head amy-why-3459 / cursor[bot] / steward 条件
linyueqian(谢谢,复算全部成立)
另外 Codex CLI(gpt-6.1-sol,xhigh)对新 head 做了一轮只读对抗评审,结论和处理见 PR 正文。请在新 head 上复核。 |
amy-why-3459
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 016c7c7df023944f0f46d8e777d6d31f1df1764c and head ac0510b21b887e34b7357b13dc3d2eb7c9e86c5f(draft)。
上次在 1b2b4593cb72ef7b956f82f000313a9910cffbd7 上的阻塞已处理:
- #454 已附条件接受(steward 评论 6897439035)。
- 已 rebase 到当前 main
016c7c7d(ahead 8 / behind 0)。 - 两开关优先级与「点估计未过、区间过门槛」已写入
docs/optimization-workflow-unification.md和 #454,并有test_both_switches_apply_the_decision_before_the_explanation、test_paired_decision_passes_when_only_the_point_estimate_misses_the_gate覆盖。
本轮还核对了 linyueqian 在旧 head 上的意见及后续修订:精确次序统计量区间、MIN_PAIRS=6、Bonferroni 家族校正、跨门槛记 INCONCLUSIVE(within_noise)、--decision paired 时 report_only: false,以及 profile 报告失败不改变默认判定。我用 n=1…40 的二项尾计算复核了 interval_rank,与文档中的对数要求(k=1 为 6,k=2–3 为 7,k=4–6 为 8,k=7–12 为 9)一致。declarations 已通过。我没有重跑 tools/measurement_simulation.py 的仿真,也没有跑 hosted 全量 pytest。
david6666666
left a comment
There was a problem hiding this comment.
One issue with how normalized evidence affects the explanation gate.
| [v / k for v, k in zip(c, clock[1])], | ||
| confidence=confidence) | ||
| entry.update(normalized=normalized, observed=normalized["observed"], | ||
| basis="normalized_to_reference_regions") |
There was a problem hiding this comment.
Could a sign reversal between raw and normalized evidence stay inconclusive unless the mechanism explicitly predicts a relative effect? This overwrites the raw direction. In a 10-pair check, the raw region interval shows a 20% increase, the normalized interval shows a decrease, and the reference interval merely crosses 1. The result is explained for an absolute decrease. Please preserve that disagreement in the explanation gate and add a regression test.
There was a problem hiding this comment.
Thanks, agreed and fixed in 6a3c1236. When a region's raw and reference-normalized intervals point in opposite definite directions, the evidence now records observed: "raw_normalized_conflict" and keeps both raw and normalized. That check's outcome is unverifiable, so explanation.status is unverifiable (INCONCLUSIVE under --require-explanation). The normalized sign no longer wins. The mechanism format has no way to declare a relative-only effect, so a reversal is never read as either direction.
A raw no-clear-change with a definite normalized direction is still resolved by normalization; that's the drift it is meant to cancel. A definite raw direction with no clear normalized change stays not_shown.
Regression: test_raw_and_normalized_sign_reversal_is_not_read_as_either_direction reproduces your 10-pair case. The raw target is +20%; two references each cross 1 in different pairs, so neither is contradicted, yet their sum rises in every pair. The normalized result is a decrease. Before the fix this was explained; now it is unverifiable. test_normalization_still_resolves_a_raw_change_hidden_by_drift keeps the intended normalization path. Codex also built a single-reference variant and checked 8,000 direction combinations without finding another reversal.
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 016c7c7 and head ac0510b (draft).
Re-review after author push from prior CHANGES_REQUESTED on 1b2b459. Prior blockers cleared on this head:
- Governing #454 acceptance already recorded.
- Rebased onto 016c7c7 with focused harness tests reported in the PR body.
- Dual-switch priority and point/interval disagreement documented in docs/optimization-workflow-unification.md and covered by test_both_switches_apply_the_decision_before_the_explanation plus test_paired_decision_passes_when_only_the_point_estimate_misses_the_gate (agrees_with_point_gate is False). Defaults-off report-only path retained.
Merge process (not content blockers on this draft head): still draft; head is behind current main again after later merges — rebase before marking ready; run ready-lane CI on the rebased tip. Hardware A/A remains non-blocking while switches default off.
Acting steward re-review while hosted Dong-bot is on usage_limit; formal Approve as cursor[bot].
…ithin noise The percentile bootstrap lower bound for a median of n pairs is an order statistic whose one-sided error exceeds 2.5% at small n (3.1% at 5 pairs, 6.25% at 7). Use the exact distribution-free order-statistic interval (the inverted sign test) instead: the rank keeps each tail at or below 2.5%, so the minimum is six pairs and A/A at the gate passes at most 2.5% of the time. Declared expectations are judged as a family: region and counter intervals use the Bonferroni confidence 1 - 5%/k, and too few pairs for it is unverifiable. A declared-unchanged item is reported as not contradicted, never confirmed. A paired interval that straddles the gate shows only that passing was not demonstrated, which is not evidence that the gate was missed, so the round is INCONCLUSIVE (within_noise) and still does not block the search. Records say report_only: false when the interval decides the verdict. Reported by linyueqian on InferMatrixLab#457. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
When a region's raw and reference-normalized intervals point in opposite definite directions, an absolute increase can read as a relative decrease (two references can each cross 1 while their sum rises in every pair). No declaration predicts a relative effect, so record raw_normalized_conflict and leave that check unverifiable instead of letting the normalized sign win. Under --require-explanation, only contradicted evidence fails. An unsupported prediction is missing evidence, like a paired interval within noise, so it is INCONCLUSIVE; the candidate is still not adopted and the search continues. Reported by david6666666 on InferMatrixLab#454 and InferMatrixLab#457. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
6a3c123
ac0510b to
6a3c123
Compare
|
New head It also addresses david6666666's two points:
The 34 related test files and ruff pass on this head. A Codex review of the change found no code issues; its only item was syncing the PR body, which is now done. Please re-review on |
amy-why-3459
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 6c7e237935a22cc5e90e954d4f397376e753e7d8 and head 6a3c123691b0ba6937084837c62a888b9735f59b(draft)。
对照上次已通过的 ac0510b21b887e34b7357b13dc3d2eb7c9e86c5f,本轮增量主要是 6a3c123691b0ba6937084837c62a888b9735f59b,并已 rebase 到 main 6c7e2379(含 #461,--decision / --require-explanation / --mechanism-file 现位于 src/inferrsi/apps/cli/arguments.py)。
已核对 david6666666 的两点:
--require-explanation下只有contradicted记 FAIL,unsupported/unverifiable/not_declared记 INCONCLUSIVE(src/inferrsi/harness/project.py与tests/test_project_explanation.py)。- raw/normalized 方向冲突记为
raw_normalized_conflict,_check_outcome将其判为unverifiable;回归测试test_raw_and_normalized_sign_reversal_is_not_read_as_either_direction覆盖了所述 10 对情形,test_normalization_still_resolves_a_raw_change_hidden_by_drift保留了归一化抵消漂移的路径。文档与 #454 已同步。
合并流程(非本 head 内容阻塞):仍为 draft;相对当前 main 2cbfab66b00d75e69be583ebfeabee2b8f83c4ea(#470/#472)落后 2 个提交,但无文件重叠,GitHub 显示可合并无冲突。标记 ready 前请再 rebase。我没有重跑作者声称通过的相关测试和 ruff,也没有跑 hosted 全量 pytest。
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 6c7e237935a22cc5e90e954d4f397376e753e7d8 and head 6a3c123691b0ba6937084837c62a888b9735f59b (draft).
Re-review after author push from prior PASS on ac0510b21b88 (Approves dismissed by rebase). Incremental head addresses david6666666: raw/normalized definite sign reversal records raw_normalized_conflict → check unverifiable; under --require-explanation only contradicted is FAIL and unsupported/unverifiable/not_declared stay INCONCLUSIVE. Prior blockers remain cleared (#454 accepted; dual-switch priority + point/interval disagreement documented/tested). Still draft; behind current main by 2 non-overlapping commits — rebase before ready. Formal Approve as cursor[bot].
|
Re-checked on
The four changed test files pass on this head (160 passed). One new gap. |
…ithin noise The percentile bootstrap lower bound for a median of n pairs is an order statistic whose one-sided error exceeds 2.5% at small n (3.1% at 5 pairs, 6.25% at 7). Use the exact distribution-free order-statistic interval (the inverted sign test) instead: the rank keeps each tail at or below 2.5%, so the minimum is six pairs and A/A at the gate passes at most 2.5% of the time. Declared expectations are judged as a family: region and counter intervals use the Bonferroni confidence 1 - 5%/k, and too few pairs for it is unverifiable. A declared-unchanged item is reported as not contradicted, never confirmed. A paired interval that straddles the gate shows only that passing was not demonstrated, which is not evidence that the gate was missed, so the round is INCONCLUSIVE (within_noise) and still does not block the search. Records say report_only: false when the interval decides the verdict. Reported by linyueqian on InferMatrixLab#457. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
When a region's raw and reference-normalized intervals point in opposite definite directions, an absolute increase can read as a relative decrease (two references can each cross 1 while their sum rises in every pair). No declaration predicts a relative effect, so record raw_normalized_conflict and leave that check unverifiable instead of letting the normalized sign win. Under --require-explanation, only contradicted evidence fails. An unsupported prediction is missing evidence, like a paired interval within noise, so it is INCONCLUSIVE; the candidate is still not adopted and the search continues. Reported by david6666666 on InferMatrixLab#454 and InferMatrixLab#457. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
With --require-explanation, fewer than six repetitions give no interval for any expectation, so every round would be unverifiable and nothing could be adopted; campaign start now rejects that. A round whose declaration needs more pairs than the frozen repetitions (Bonferroni over its expectations) is rejected before execution, so it spends no budget. Without the switch a declaration remains a report and is not restricted. Reported by linyueqian on InferMatrixLab#457. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
60ba2d0
6a3c123 to
60ba2d0
Compare
|
Thanks for the re-check, and agreed on the new gap. Fixed in
The rebase onto |
david6666666
left a comment
There was a problem hiding this comment.
One case where the paired verdict and campaign selection disagree.
| "completed", "fixed_checks_satisfied" if accepted else "performance_gate_not_met", | ||
| verdict="PASS" if accepted else "FAIL", score=after, baseline_score=before, | ||
| "completed", reason, | ||
| verdict=verdict, score=after, baseline_score=before, |
There was a problem hiding this comment.
In --decision paired mode, this still hands the selector the unpaired median. In a nine-pair minimize fixture at a 1% gate, the paired interval was about [1.111, 1.111] (PASS, eligible=true), but medians 6 vs 9 left the baseline selected, so final never validated the passing candidate. Can we define whether paired evidence controls adoption as well as the verdict and add a campaign-level test for this disagreement? If point-score ranking is intentional, please document the veto and report why the passing round was not selected.
There was a problem hiding this comment.
Thanks, confirmed and fixed in 507ce154 with your option A (the branch is also rebased onto current main 0311393c). I reproduced it exactly: baseline [1,2,3,4,6,10,10,10,10], candidate [0.9,1.8,2.7,3.6,100,9,9,9,9]. That gives the interval [1.111, 1.111] and a PASS, with medians 6 vs 9, and _settle kept the baseline. Paired evidence now controls adoption as well as the verdict.
-
Frozen selection. Under
--decision paired,campaign startfreezesselection = {"metric": "paired_median_speedup", "direction": "maximize", "initial_score": 1.0}in the digest-bound plan, so the basis can't change mid-search. Each round'sscore(andspeedup) is its median paired speedup against the frozen baseline, and the baseline counts as 1.0. A candidate is adopted when it isPASSand that speedup beats the current best. Every round re-measures the frozen baseline in alternating order, so speedups from different rounds are all relative to the same baseline and comparable. -
No controller special case. The generic
_settlecomparison is unchanged. Paired mode reuses the samemaximize/initial_score: 1.0selection shape the benchmark sessions already use, and defaultpointmode and benchmark sessions select exactly as before. -
Headline numbers follow the decision.
improvement_pctis derived from the paired speedup. The unpaired values remain inpoint_estimate, and the raw medians remain inmetrics, sosummary.optimization_speedupafter final is the paired estimate. Before, a selected paired PASS would have been reported as 0.667×. If the paired statistics are unavailable (for example, a paired ratio underflows even though every sample is finite and positive), the round is INCONCLUSIVE. Itsscore,speedupandimprovement_pctare left empty rather than filled with point values relabelled as paired, and the round is not adopted. The controller accepts a completed round without a score only when it is explicitlyeligible: false, so the search continues instead of blocking.test_unavailable_paired_statistics_neither_block_nor_borrow_a_point_scorecovers this for a round and for final. -
Why a round was not selected. For the reporting half of your question: every round record now carries
selection: {outcome, metric, score, best_score_before}.outcomeis one ofadopted,not_eligible,not_better_than_bestorbaseline_identity. A PASS that is outranked by an earlier, faster candidate is recorded asnot_better_than_bestrather than left implicit. This is additive and applies in both modes. -
Extreme values settle. In paired mode, a point value that can't be represented as a finite float (an overflowing median, improvement or speedup) is recorded as null instead of failing the round, because the decision doesn't read it. If the paired median itself is unavailable, the round is INCONCLUSIVE and recorded, as above. Default
pointmode still treats a non-finite point estimate as an infrastructure failure.
Campaign-level regression: test_paired_decision_also_selects_the_candidate_that_final_retests runs your nine-pair minimize fixture at the 1% gate through real campaign start/round/final:
- Round 1 shows medians 6/9, a point estimate of −50% and paired PASS at 1.111. It is adopted, with
best_score_before1.0. - Round 2 is a 1.05× PASS. It is recorded as
not_better_than_best. finalretests the round-1 candidate (PASS), andoptimization_speedupis 1.111.- The managed dashboard still labels the medians with the measured metric (
metric, ms, minimize) and shows the round as adopted at +10%.
Documented in docs/optimization-workflow-unification.md and the --decision help.
zengchuang-hw
left a comment
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
独立检查 base bebd491 / head 60ba2d0。本次以 COMMENT 提交可复现问题,不作批准。
[P2] paired 判定与 controller 选优仍使用不同的改善标准
位置:src/inferrsi/harness/project.py:225-228;调用方 src/inferrsi/harness/optimization.py:275-285。
paired 模式可以返回 PASS / eligible=true,但 score 仍是候选的独立中位数。controller 随后又要求该 score 优于 baseline/best 的独立中位数,因此已通过配对门槛的首个候选仍可能被丢弃,final 将重测 baseline 而非该候选。
已用真实 campaign start/round 路径复现:10 次测量、minimize、1% 门槛、decision=paired;baseline 测量值为 11..20,candidate 逐项除以 1.03,仅把第一项改为 200。九对加速比是 1.03,配对区间高于门槛,assessment 为 PASS/eligible=true;但 baseline_score=15.5、score=16.019417475728154、improvement_pct=-3.3510804885687384,state.best.kind 仍为 baseline。现有“不通过点门槛但通过 paired”测试只覆盖独立中位数仍有正改善的情况,未覆盖这里的符号反转。
请使 paired 模式的候选选择与其声明的配对判定语义一致,至少补充上述首个合格候选及 final 选择的集成测试;若保留独立中位数作为第二层筛选是有意的,应明确记录这一额外筛选及 rejection 原因,而不能把该模式表述为只看配对区间。
实际验证:本轮重新执行 tests/test_harness_measurement.py、tests/test_project_explanation.py、tests/test_project_optimization.py、tests/test_optimization_session.py,共169项通过;ruff check src tests tools 与 git diff --check 通过。额外独立回归用例确认上述不采纳行为(期望选中候选的断言失败)。合成消融工具以 campaigns=20 / resamples=100 跑完70组,仅作为软件烟测,不作统计精度或硬件性能证明。
未执行:完整仓库测试、完整默认规模消融、GPU/NPU或服务实测、精确head的远端CI验证。需求核验限制:已获得隔离的功能范围,但未取得可独立溯源的原始统计方法、状态映射、重复次数与开关优先级细则,因此不声称完整验收通过。
Pair the k-th baseline and candidate samples that steps() already alternates, and report the median paired speedup with a seeded percentile bootstrap interval and its relation to the frozen gate. The field is report-only: verdict, eligibility and selection are unchanged, and any value that cannot be computed is reported as unavailable so records stay strict JSON. Fewer than five pairs report no interval because simulated coverage at three pairs is about 75%. tools/measurement_simulation.py reproduces the synthetic ablation. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Agent-assisted-by: Codex CLI / gpt-6.1-sol [effort=high] (review) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
…ches Measurement scripts may add per-step region breakdowns and work counters. A round may declare its mechanism and expected region/counter directions. Regions declared unchanged act as an in-step clock, so ratios to them and their share of the step total cancel device-wide clock or thermal drift; counters such as launches or DRAM bytes show whether a small gain is reproducible. Directions come only from paired intervals, with no repository-wide tolerance. Two switches are frozen at campaign start and default to the existing behaviour: --require-explanation adopts only candidates whose mechanism is confirmed by the data, and --decision paired requires the paired interval to clear the gate, failing within-noise rounds without blocking the search. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
… evidence Bind the round's mechanism into control state before the candidate runs and assess against that copy, so a candidate cannot rewrite its own declaration. Require five pairs before any profile direction, treat zero-baseline ratios as undefined, report unavailable shares instead of raising, keep raw evidence for every region, and never normalize declared references against each other. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Agent-assisted-by: Codex CLI / gpt-6.1-sol [effort=xhigh] (review) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
The candidate runs with the caller's privileges and can write state.json or the attempt copy. Pass the declaration held since submission to the assessor, record its digest, and treat a copy that no longer matches as an integrity failure. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Agent-assisted-by: Codex CLI / gpt-6.1-sol [effort=xhigh] (review) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
An interrupted attempt is settled by a new process that can only read files the candidate could write. Settle it without the declaration, mark the record, and leave require_explanation to keep the round unadopted. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Agent-assisted-by: Codex CLI / gpt-6.1-sol [effort=xhigh] (review) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
With both switches on, --decision sets the performance verdict first and --require-explanation only downgrades a PASS. The paired decision reads the interval alone, so a candidate whose point estimate misses the gate but whose paired interval clears it passes, with agrees_with_point_gate recorded false. Regression tests pin both behaviors. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
…ithin noise The percentile bootstrap lower bound for a median of n pairs is an order statistic whose one-sided error exceeds 2.5% at small n (3.1% at 5 pairs, 6.25% at 7). Use the exact distribution-free order-statistic interval (the inverted sign test) instead: the rank keeps each tail at or below 2.5%, so the minimum is six pairs and A/A at the gate passes at most 2.5% of the time. Declared expectations are judged as a family: region and counter intervals use the Bonferroni confidence 1 - 5%/k, and too few pairs for it is unverifiable. A declared-unchanged item is reported as not contradicted, never confirmed. A paired interval that straddles the gate shows only that passing was not demonstrated, which is not evidence that the gate was missed, so the round is INCONCLUSIVE (within_noise) and still does not block the search. Records say report_only: false when the interval decides the verdict. Reported by linyueqian on InferMatrixLab#457. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
A float 2 ** pairs overflows near a thousand pairs; compare binomial tails as exact rationals instead. A failure while building profile evidence is reported as unavailable rather than raised, so it cannot change the default verdict. Found by Codex review of the exact-interval change. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Agent-assisted-by: Codex CLI / gpt-6.1-sol [effort=xhigh] (review) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
When a region's raw and reference-normalized intervals point in opposite definite directions, an absolute increase can read as a relative decrease (two references can each cross 1 while their sum rises in every pair). No declaration predicts a relative effect, so record raw_normalized_conflict and leave that check unverifiable instead of letting the normalized sign win. Under --require-explanation, only contradicted evidence fails. An unsupported prediction is missing evidence, like a paired interval within noise, so it is INCONCLUSIVE; the candidate is still not adopted and the search continues. Reported by david6666666 on InferMatrixLab#454 and InferMatrixLab#457. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
With --require-explanation, fewer than six repetitions give no interval for any expectation, so every round would be unverifiable and nothing could be adopted; campaign start now rejects that. A round whose declaration needs more pairs than the frozen repetitions (Bonferroni over its expectations) is rejected before execution, so it spends no budget. Without the switch a declaration remains a report and is not restricted. Reported by linyueqian on InferMatrixLab#457. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
Under --decision paired the verdict read the paired interval, but adoption still compared independent candidate medians, so a paired PASS could be outranked and never reach final. The frozen selection is now the median paired speedup against the baseline (1.0), and that same speedup is the round's score, speedup and improvement. Point values stay in point_estimate. Each round records why it was or was not selected. The default point mode and benchmark sessions keep their selection. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: BeatSeat <137616995+BeatSeat@users.noreply.github.com>
60ba2d0 to
507ce15
Compare
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent incremental review, COMMENT: 60ba2d0 → 507ce15, current base c6678fa. Previous independent result: #457 (review). I did not use other review opinions or author-result narratives.
The previously reported paired-selection mismatch is addressed in the inspected code:
- project.assess now retains independent-median measurements separately as point_estimate and, for paired decisions, supplies median paired speedup as score with baseline_score=1.0.
- Project session initialization freezes paired selection as maximize / paired_median_speedup / initial_score=1.0, rather than feeding a raw latency objective to the controller.
- The controller records selection outcome and its comparison basis. Final execution still takes the selected candidate path; new source-level regression coverage explicitly checks that the paired candidate is selected and independently retested at final. I inspected those tests but did not run them here.
Fresh execution was deliberately bounded to read-only, in-memory Windows/Python probes because the cloud test workspace is offline:
- Executed the exact-head paired_statistics implementation and assess function with synthetic step-reader/profile stubs. Reused my previous counterexample: baseline11..20, candidate baseline/1.03 with the first value replaced by200. Result: PASS, eligible=true, score1.03, baseline_score1.0, score_basis paired_median_speedup; the independent point improvement remains -3.3510804885687384%. Thus the negative point estimate is no longer the selection objective.
- Executed the exact current controller round-selection block in memory with synthetic state and candidate-identity stub. It adopts that score1.03 candidate against baseline1.0.
These two probes passed. They are not pytest results, a full campaign start/round/final integration run, or a statistical/hardware validation. No repository or campaign files were written.
Machine evidence: independently inspected declarations jobs134155137709/run44116293063 and134153622318/run44115867662. Both evaluate exact head507ce154bc80e68e0a303e26af8a4db79f4313d4 with policyc6678fa8df0e2ab423587eba06151e8386b5b9ce and report compliant/job_exit0. These are declaration-policy results, not product-test passes. The observed current-head check list contained those declaration checks; I have no current-head full-suite/installed/browser pass to certify.
Remaining verification: rerun the new paired-selection/final integration tests and relevant measurement/project/session regression modules on a complete checkout, then inspect applicable current-head CI. I did not rerun pytest, Ruff, diff-check, complete ablation, installed packages, or real GPU/NPU/service work this round. The original statistical-method/state-mapping/repetition/switch-precedence acceptance provenance remains incomplete; I do not silently treat the new implementation or tests as its accepted specification. Other incoming-main changes were not certified by this focused review.
Conclusion: the old mismatch has direct code and bounded executable evidence of correction, but end-to-end and original-AC gaps remain. COMMENT, not APPROVE; no merge, code edit, experiment restart or deployment.
Governing Issue and delivery slice
Refs #454
campaign start/round/final)新增三项能力,默认都不改变现有判定:steps()已有的 AB/BA 交替,把同一次重复里的基线和候选样本配对,报告逐对加速比中位数的精确次序统计量区间(符号检验反演,至少 6 对),以及区间与门槛的关系。breakdown(分区域耗时)和counters(工作量计数);每轮可以用--mechanism-file声明机制和预期方向;Harness 按声明条数做 Bonferroni 校正后机械核对,给出每条核对的outcome和explanation.status。--require-explanation只采纳explained的候选;--decision paired用配对区间判定,并按逐对加速比中位数选优(未配对中位数不能否决配对 PASS);区间跨门槛判 INCONCLUSIVE(within_noise),不被采纳也不阻塞搜索。两者同时打开时,先由--decision得出性能结论,--require-explanation只降级 PASS。benchmarks session拒绝--mechanism-file);Dashboard 展示;与具体 profiler 绑定的采集脚本;GPU/NPU 实测。Knowledge authorship declaration
本 PR 不触及 knowledge-model policy 的受保护路径,预期为 Not applicable。
Changed artifacts and evidence
src/inferrsi/harness/measurement.pyinterval_rank、median_interval、min_pairs,二项尾用有理数精确比较)、配对统计、profile_evidence(Bonferroni)(原始与归一化方向相反时记为raw_normalized_conflict)、validate_mechanism、explain;无法计算的值一律返回unavailable,记录保持严格 JSONsrc/inferrsi/harness/project.pyassess()接入两个开关和机制核对;--decision paired时score/speedup/improvement_pct取逐对加速比中位数(基线 1.0),点估计保留在point_estimate,不可表示的值记为空而不阻断;profile 报告出错时记为unavailable,不影响判定;新增设置项的校验(--require-explanation至少 6 次重复)src/inferrsi/harness/optimization.pyselection(adopted/not_eligible/not_better_than_best/baseline_identity);明确eligible: false的完成轮次可以没有 scoresrc/inferrsi/harness/optimization_session.pypaired时把选优冻结为paired_median_speedup(maximize,初值 1.0);读取并校验--mechanism-file,开关打开时在执行前拒绝所需对数超过冻结重复次数的声明;_runtime_identity纳入measurement.pysrc/inferrsi/apps/views/dashboard_data.pymetrics.name/direction)标注中位数,不用选优指标src/inferrsi/apps/cli/arguments.py--decision、--require-explanation、--mechanism-filedocs/optimization-workflow-unification.mdpaired下的选优与selection记录、局限tests/test_harness_measurement.pyunchanged只记"未推翻"、原始与归一化方向相反不被读成任一方向、机制校验与分类tests/test_project_explanation.pynot_better_than_best;配对统计不可用或点估计溢出时记录轮次而不阻断;确认的机制在 final 复核;不可采纳情形都让搜索继续;开关关闭时只报告;within_noise为 INCONCLUSIVE;两开关组合 6 例;点估计未过而区间过门槛判 PASS;profile 报告出错不改判;重复次数不足时 start 拒绝、声明无法核对时在执行前拒绝且不耗预算、开关关闭时仍只报告;篡改state.json或副本;中断后伪造声明的恢复tests/test_project_optimization.py、tests/test_optimization_session.pytools/measurement_simulation.py--gate-pct 0给出门槛处 A/A 比例与精确上界判定逻辑
explanation.status--require-explanation下explainedconfirmed,且没有任何一条contradictedunsupportednot_shown(未检出;区间仍可能包含预期的变化)contradictedunchanged的项检出变化unverifiablenot_declared--decision paired区间与门槛above_gatepaired_interval_above_gate(即使点估计未过门槛;此时agrees_with_point_gate: false)straddles_gatewithin_noise(没能证明过门槛,不是门槛未达到的证据)below_gate、gate_unreachableperformance_gate_not_metpaired_statistics_unavailable,不退回未配对规则在
--require-explanation下只有contradicted(数据与预期相反)记 FAIL,证据不足的情形都记 INCONCLUSIVE,与配对性能判定的原则一致。若某区域原始比较和"除以参照区域"后的比较给出明确相反的方向(例如原始增加 20%、相对参照却下降),记为raw_normalized_conflict,该条核对为unverifiable:机制声明无法表达"只预期相对变化",两个方向都不采信。声明为
unchanged的项只会被记为not_contradicted(未检出变化)或contradicted,不会被"确认不变":没有等价性容差时,区间包含 1 不是"没变"的证据。声明为unchanged的区域同时是本轮的假设:其他区域会另外给出"除以同一步骤参照区域"后的比较,share_based_speedup用参照占比推算整体加速。只有让一个步骤内所有区域按同一比例缩放的变化,才会被这样抵消。参照区域本身只做原始比较,不会互相归一化。合成消融(
python tools/measurement_simulation.py,CPU 仿真,非设备实测)每格 200 次 campaign,数据 seed 20261115;不配对区间这一对照规则用 bootstrap 10000 次。200 次 campaign 的 Monte Carlo 标准误约 0.01–0.03,门槛处的误报以下方精确上界和大样本复核为准。
Checks and dependencies
ruff check src tests tools:通过。bebd4914)上运行 36 个 harness / campaign / project / dashboard / optimization 相关测试文件(含tests/test_harness_measurement.py、tests/test_project_explanation.py、tests/test_project_optimization.py、tests/test_optimization_session.py、tests/test_module_boundaries.py、tests/test_target_module_import_gate.py、tests/test_package_root_inventory.py和 Dashboard 测试):全部通过,无失败。Git 相关测试使用临时 Git 2.49.1(本机 2.43 不支持--no-lazy-fetch)。gpt-6.1-sol,xhigh)本地只读评审,每一轮都用对抗输入去复现问题。Infinity;零基线计数在 1 对时就给出确定方向;breakdown 全为零时抛异常;多个参照区域互相归一化;候选在执行期间改写mechanism.json或state.json中的声明;中断后经recover接受伪造的声明(现记录mechanism_unverified_after_recovery,按未声明处理)。2 ** n转浮点溢出,且 profile 报告异常会改变默认判定。已修复(有理数精确比较;报告出错记为unavailable)并补回归测试。第八轮复核结论:Ready(1024 对复现通过;注入报告异常后默认判定不变;31,255 组精确秩检查无偏差)。unverifiable;8,000 组方向组合检查无反向误判;原始样本不足或零基线不会经归一化变成明确方向;assess 与独立 final 都是unsupported → INCONCLUSIVE、contradicted → FAIL,搜索继续;rebase 后无旧 CLI 路径残留。结论 Ready with fixes,唯一一项是同步本 PR 说明(已同步)。--require-explanation下 start 拒绝 3/5 次重复、round 在执行前拒绝 6 次重复下的 4 条预期,返回用法错误,campaign 不变、不耗轮次;k = 1…12 的所需对数与pairs_needed完全一致;开关关闭时仍可运行。结论 Ready,无 P0/P1/P2。plan.json已有plan_digest保护。measurement.py只用标准库。steps()两侧等长且交替执行;区间的精确性依赖各次重复独立、比值来自连续分布,离散计数出现并列时偏保守。harness/*.py会改变 runtime identity,旧 runtime 启动的会话需要用原来冻结的 runtime 恢复(既有规则)。评审后的修订
0311393c(含 #461 的 CLI 迁移、#474、#323、#477、#466、#483、#475)。optimization.py的_settle与 main891aadda冲突:保留 main 新增的"执行必须正常结束"检查,其后用执行前绑定的声明调用判定器;新 head 上重跑测试test_both_switches_apply_the_decision_before_the_explanation(6 例)、test_paired_decision_passes_when_only_the_point_estimate_misses_the_gateorder_statistic_rank、achieved_confidenceunchanged不是证据,且没有多重比较校正family_size、confidence、pairs_needed);unchanged只记not_contradicted/contradicteddocs/contributing.md的 FAIL 定义冲突within_noise);低于门槛仍为 FAILreport_only: true--decision paired时为falseraw_normalized_conflict,该条核对unverifiable;test_raw_and_normalized_sign_reversal_is_not_read_as_either_direction复现了他描述的 10 对情形(两个参照各自跨过 1,但总和每对都上升),修复前判explained,修复后为unverifiableunsupported是证据不足,应与contradicted区分--require-explanation下unsupported改为 INCONCLUSIVE,只有contradicted记 FAIL;候选都不被采纳--require-explanation在少于 6 次重复时仍可启动,任何预期都拿不到区间,没有候选能被采纳;单轮声明所需对数超过冻结重复次数时也注定unverifiablecampaign start拒绝少于 6 次重复;round --mechanism-file在执行前拒绝所需对数(与profile_evidence的pairs_needed同一公式)超过冻结重复次数的声明,不记录轮次、不耗预算;开关关闭时声明仍只报告--decision paired下判定看配对区间,选优仍比较未配对中位数:9 对、1% 门槛时区间 [1.111, 1.111] 判 PASS,但中位数 6 vs 9 使基线仍被选中,final 没有复测该候选selection = paired_median_speedup(maximize,基线 1.0),score/speedup/improvement_pct取逐对加速比中位数,未配对值留在point_estimate;每轮记录selection.outcome说明是否被选中及原因;默认point与题包会话的选优不变。test_paired_decision_also_selects_the_candidate_that_final_retests端到端复现他的 9 对情形(采纳、final 复测、optimization_speedup1.111)Remaining work and Issue closure
Remaining Issue requirements / follow-up:
hypothesis、verdict 和采纳结果;mechanism、explanation(含每条outcome)、profile_evidence、paired_statistics已在同一份控制记录里,后续可在 Apps 视图层只读投影,不另写记录;另有互补的知识 PR(task: add GPU measurement-validity and profiler-attribution guidance to inference-benchmark-design #455),两者互不依赖。
Goal-acceptance owner or accepted closure delegation: Dong1017(rfc: 可解释的改进——用 profile 证据和配对统计判定 PASS #454 记录);未授予关闭委托。
Goal ready for acceptance? Not claimed