Repository navigation
Open
Labels
area:harnessHarness module: profile, A/B loop, quality gates, evidence reportsHarness module: profile, A/B loop, quality gates, evidence reportskind:acceptanceCampaign acceptance issue: frozen gates + evidence tree; PASS/FAIL/INCONCLUSIVECampaign acceptance issue: frozen gates + evidence tree; PASS/FAIL/INCONCLUSIVE
Description
Refs #337
Feature delivery: #501
Campaign and pinned scope
- Campaign: RSI managed Knowledge on/off, eight-round qualification.
- Framework baseline:
cf671a880ac8037e01922e1be0b701771fb9922f(PR base). - Framework candidate and shared runtime pin for both arms:
9772e690801552402b71d84f9e08368814a1e5ff. - Experimental treatment: managed Knowledge on/off. Both arms use the candidate runtime, the same frozen task-specific baseline and validator, and independent native Agent sessions. The CLI records task and source digests.
- RunSpec:
InferMatrixLab/InferRSI, Ascend 910B1, Qwen denoise on one NPU per arm, MoE forward on two NPUs per arm. Same-task on/off sessions overlap with separate device allocations. Requested and observed Agent settings: GPT-6.2 Sol / medium. The two tasks execute in sequence. - Budget: eight optimization rounds plus one independent final per arm, one paired replicate per task, 240 campaign minutes per arm. Retain timing, token coverage, process cleanup, and allocation release evidence.
Frozen acceptance criteria
- Run both complete task suites through the normal compare/dispatch/collect/report path.
- Freeze identical baseline, workload, sampling and correctness protocol within each pair. Use the packaged task validators, including both MoE ranks.
- Each arm records eight attempted optimization rounds and an independent final tied to the selected candidate.
- Verify actual model, effort and device profile, candidate binding and the evidence hashes.
- Generate HTML, Markdown and JSON with paired results, optimization traces, independent final scores, timing, tokens and gaps.
- Retain infrastructure failures. After diagnosis, rerun an affected pair under the same protocol rather than mixing execution conditions.
There is no service latency or minimum improvement gate. The acceptance decision covers the benchmark workflow. The on/off results are descriptive observations tied to the tasks, budget, Agent settings and one paired replicate.
Evidence and status
INCONCLUSIVE until the formal run and evidence verification are complete. The comparison freezes the plan and hashes in its experiment directory, retains each arm's campaign tree and emits the three report files together. The sanitized summary and tested scope will be posted to #501.
Metadata
Metadata
Assignees
Labels
area:harnessHarness module: profile, A/B loop, quality gates, evidence reportsHarness module: profile, A/B loop, quality gates, evidence reportskind:acceptanceCampaign acceptance issue: frozen gates + evidence tree; PASS/FAIL/INCONCLUSIVECampaign acceptance issue: frozen gates + evidence tree; PASS/FAIL/INCONCLUSIVE
Activity
Dong1017 commentedon Nov 18, 2026
Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).
Triage: this is a well-scoped acceptance/tracking issue for an eight-round managed Knowledge on/off benchmark. It is currently INCONCLUSIVE by its own statement, and nothing here can be verified until the formal run exists. Notes below are from reading the body only. I haven't checked the linked issues, the pinned SHAs, or any run artifacts.
What it asks
Strengths
cf671a88…) and the candidate/runtime pin (9772e690…) are pinned. Both arms use the candidate runtime, so Knowledge on/off is the only treatment.Gaps and risks
Suggested next steps
No decision was requested from Dong1017 in the body, so nothing to record here. I'll review the evidence when it's posted.
david6666666 commentedon Nov 18, 2026
Could the formal run have a pre-dispatch gate for the frozen runtime? The pinned
9772e690has failing full-check jobs and an open MoE rank-evidence finding in #501, so it is not yet a clean qualification baseline. Once the corrected head is reviewed, freeze a new plan and dispatch both arms of each pair on that one revision. Keep anything already collected on this head as preflight or historical evidence, not one half of a formal pair.zengchuang-hw commentedon Nov 18, 2026
独立检视(验收方案;静态核查,未运行实验)
范围:本 Issue 当前正文及固定 runtime
9772e690801552402b71d84f9e08368814a1e5ff;阅读了 AGENTS、contributing 验收/证据规则、#337 的任务说明,以及该 pin 的docs/benchmark-comparison.md、apps/benchmark_compare.py与benchmark/comparison.py。未读取其他人的检视意见。方案把两臂固定在相同 candidate runtime、仅切换 managed Knowledge,并把结果限定为单次配对的描述性观察,这个边界是合理的。
cf671a88…是框架 PR base,不是第三个实验臂,也不能把本实验解释为框架新旧版本的性能比较。当前仍应保持 INCONCLUSIVE,不是 workflow PASS 或 Knowledge 效果证明。验收前需要明确并随证据交付的内容:
job_max_minutes、单次 evaluation timeout、final reserve 和实际 task arguments。固定代码_validate_config要求 job 预算大于 campaign 预算,默认 job 仅 90 分钟,因此不能只覆盖max_minutes=240后沿用其他默认值。此处是运行配置完整性要求,不断言实际准备已经失败。采样、warmup、MoE 两 rank 的检查参数也应可追溯。benchmark/comparison.py的efficiency_improved/no_gain/regression/tradeoff/insufficient_evidence是配对比较结论,不是仓库的 PASS/FAIL/INCONCLUSIVE 验收判定。本 Issue 没有 minimum improvement gate,因此完整、合规的无收益或负收益实验不应仅因收益方向被判 workflow FAIL;反过来,有正向数字也不能覆盖缺 final、环境不匹配或缺失配对。_trial_result/report会复核 campaign,实际 model/effort/environment 仍来自 caller-exported observation;需保留其底层回执,不能仅以请求值或手填 fingerprint 证明环境合格。未知 token 保持 unknown,并注明计数覆盖范围。已执行:通过连接器核实 Issue/仓库身份、读取上述方案与固定源码,发表前核对本人无既有评论。未执行:仓库代码、测试、真实 Agent/NPU 运行、原始 campaign archive 哈希验证或硬件资格验证;本意见不宣称这些通过,也不触发设备实验。待证据齐全后,按上述冻结范围做验收增量核查。