Skip to content

acceptance: eight-round RSI managed Knowledge comparison #502

Open

Description

@Bounty-hunter

Refs #337
Feature delivery: #501

Campaign and pinned scope

  • Campaign: RSI managed Knowledge on/off, eight-round qualification.
  • Framework baseline: cf671a880ac8037e01922e1be0b701771fb9922f (PR base).
  • Framework candidate and shared runtime pin for both arms: 9772e690801552402b71d84f9e08368814a1e5ff.
  • Experimental treatment: managed Knowledge on/off. Both arms use the candidate runtime, the same frozen task-specific baseline and validator, and independent native Agent sessions. The CLI records task and source digests.
  • RunSpec: InferMatrixLab/InferRSI, Ascend 910B1, Qwen denoise on one NPU per arm, MoE forward on two NPUs per arm. Same-task on/off sessions overlap with separate device allocations. Requested and observed Agent settings: GPT-6.2 Sol / medium. The two tasks execute in sequence.
  • Budget: eight optimization rounds plus one independent final per arm, one paired replicate per task, 240 campaign minutes per arm. Retain timing, token coverage, process cleanup, and allocation release evidence.

Frozen acceptance criteria

  • Run both complete task suites through the normal compare/dispatch/collect/report path.
  • Freeze identical baseline, workload, sampling and correctness protocol within each pair. Use the packaged task validators, including both MoE ranks.
  • Each arm records eight attempted optimization rounds and an independent final tied to the selected candidate.
  • Verify actual model, effort and device profile, candidate binding and the evidence hashes.
  • Generate HTML, Markdown and JSON with paired results, optimization traces, independent final scores, timing, tokens and gaps.
  • Retain infrastructure failures. After diagnosis, rerun an affected pair under the same protocol rather than mixing execution conditions.

There is no service latency or minimum improvement gate. The acceptance decision covers the benchmark workflow. The on/off results are descriptive observations tied to the tasks, budget, Agent settings and one paired replicate.

Evidence and status

INCONCLUSIVE until the formal run and evidence verification are complete. The comparison freezes the plan and hashes in its experiment directory, retains each arm's campaign tree and emits the three report files together. The sanitized summary and tested scope will be posted to #501.

Activity

  1. Dong1017 commented on Nov 18, 2026

    @Dong1017
    Collaborator

    Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

    Triage: this is a well-scoped acceptance/tracking issue for an eight-round managed Knowledge on/off benchmark. It is currently INCONCLUSIVE by its own statement, and nothing here can be verified until the formal run exists. Notes below are from reading the body only. I haven't checked the linked issues, the pinned SHAs, or any run artifacts.

    What it asks

    • Run both task suites (Qwen denoise on 1 NPU per arm, MoE forward on 2 NPUs per arm) through the normal compare/dispatch/collect/report path.
    • Each pair is frozen with the same baseline, validator and protocol. Each arm gets 8 optimization rounds plus one independent final.
    • Output is HTML, Markdown and JSON reports with the evidence hashes verified. Results are descriptive only: there is no latency or minimum-improvement gate.

    Strengths

    • The baseline (cf671a88…) and the candidate/runtime pin (9772e690…) are pinned. Both arms use the candidate runtime, so Knowledge on/off is the only treatment.
    • Infrastructure failures are retained, and an affected pair is rerun under the same protocol rather than mixing conditions.
    • The claim is scoped to the tasks, budget, Agent settings and one paired replicate.

    Gaps and risks

    1. Concurrency confound. On/off sessions for the same task overlap on separate devices. Shared host, interconnect, thermal or scheduler noise is therefore not controlled. Please state how device allocation is isolated, and record per-arm device IDs and host load in the evidence.
    2. One paired replicate. Agent runs are high-variance, so the on/off difference will be hard to separate from run-to-run noise. The "descriptive" framing is right. The report should say explicitly that no on/off superiority claim is supported, and that the point of the run is workflow qualification.
    3. "Attempted" rounds vs. a valid round. The criteria say "eight attempted optimization rounds". Define what counts when a round crashes, times out, or fails the validator. Also define which candidate the independent final is "tied to" (best by validator score, last round, or Agent-selected), and whether that selection rule is frozen before the run.
    4. Rerun policy. "After diagnosis, rerun an affected pair" needs a definition of an infrastructure failure versus an Agent/task failure. Otherwise reruns can become outcome-dependent. Please also require the original failed pair to stay in the report.
    5. Budget semantics. 240 campaign minutes per arm is stated, but not what happens if rounds hit the cap before round 8. Is that a failure, a truncation, or a partial result?
    6. Agent settings. "GPT-6.2 Sol / medium" is both requested and observed. The acceptance check should state where the observed value comes from (session metadata vs. the CLI record) so it isn't self-attested.
    7. Knowledge-arm leakage. Please say how "off" is enforced and verified, for example that no managed Knowledge is retrievable in that session. Also say whether Knowledge state carries over between rounds or tasks in the "on" arm. Sequential tasks could otherwise contaminate each other.
    8. Token coverage. "Token coverage" is retained but has no pass condition. If token counts are incomplete for an arm, is that a reported gap or a failure of the acceptance?

    Suggested next steps

    • Edit the body or post a short protocol addendum covering items 1, 3, 4, 5 and 7 before the formal run starts, so the criteria stay frozen.
    • When the run completes, post to feat: benchmark RSI knowledge with paired optimization runs #501 the experiment directory hashes, the three report files, the pair-level infrastructure-failure/rerun log, and the actual tested scope. Then tick the checklist items with links to the evidence.
    • Keep the status INCONCLUSIVE until the evidence-hash verification passes. I'd also add a plain "not a performance claim" line to the final summary.

    No decision was requested from Dong1017 in the body, so nothing to record here. I'll review the evidence when it's posted.

  2. david6666666 commented on Nov 18, 2026

    @david6666666
    Collaborator

    Could the formal run have a pre-dispatch gate for the frozen runtime? The pinned 9772e690 has failing full-check jobs and an open MoE rank-evidence finding in #501, so it is not yet a clean qualification baseline. Once the corrected head is reviewed, freeze a new plan and dispatch both arms of each pair on that one revision. Keep anything already collected on this head as preflight or historical evidence, not one half of a formal pair.

  3. added
    kind:acceptanceCampaign acceptance issue: frozen gates + evidence tree; PASS/FAIL/INCONCLUSIVE
    area:harnessHarness module: profile, A/B loop, quality gates, evidence reports
    on Nov 18, 2026
  4. zengchuang-hw commented on Nov 18, 2026

    @zengchuang-hw
    Contributor

    独立检视(验收方案;静态核查,未运行实验)

    范围:本 Issue 当前正文及固定 runtime 9772e690801552402b71d84f9e08368814a1e5ff;阅读了 AGENTS、contributing 验收/证据规则、#337 的任务说明,以及该 pin 的 docs/benchmark-comparison.md、apps/benchmark_compare.py 与 benchmark/comparison.py。未读取其他人的检视意见。

    方案把两臂固定在相同 candidate runtime、仅切换 managed Knowledge,并把结果限定为单次配对的描述性观察,这个边界是合理的。cf671a88… 是框架 PR base,不是第三个实验臂,也不能把本实验解释为框架新旧版本的性能比较。当前仍应保持 INCONCLUSIVE,不是 workflow PASS 或 Knowledge 效果证明。

    验收前需要明确并随证据交付的内容:

    1. 把完整预算配置落到冻结 plan,并公布摘要/哈希。 当前正文给出 240 campaign minutes,但还应列出 job_max_minutes、单次 evaluation timeout、final reserve 和实际 task arguments。固定代码 _validate_config 要求 job 预算大于 campaign 预算,默认 job 仅 90 分钟,因此不能只覆盖 max_minutes=240 后沿用其他默认值。此处是运行配置完整性要求,不断言实际准备已经失败。采样、warmup、MoE 两 rank 的检查参数也应可追溯。
    2. 把 workflow 验收与效果方向拆开。 benchmark/comparison.py 的 efficiency_improved/no_gain/regression/tradeoff/insufficient_evidence 是配对比较结论,不是仓库的 PASS/FAIL/INCONCLUSIVE 验收判定。本 Issue 没有 minimum improvement gate,因此完整、合规的无收益或负收益实验不应仅因收益方向被判 workflow FAIL;反过来,有正向数字也不能覆盖缺 final、环境不匹配或缺失配对。
    3. 交付可复核的证据索引,而不只贴三份报告。 至少关联 frozen experiment/RunSpec 与 validator/task/source digest、四个独立 native session 的原始回执、每臂 8 次尝试和 selected-candidate final 的绑定、失败与重跑 pair 的完整索引、cleanup/allocation release 记录。_trial_result/report 会复核 campaign,实际 model/effort/environment 仍来自 caller-exported observation;需保留其底层回执,不能仅以请求值或手填 fingerprint 证明环境合格。未知 token 保持 unknown,并注明计数覆盖范围。
    4. 并行两臂的时间指标限定解释。 单独分配设备仍不证明不存在共享宿主、allocator 或其他争用。记录配对起止时间、设备分配与运行环境;将 wall time 明确称为该并发调度条件下的 campaign elapsed time,不等同纯优化计算成本或服务时延。无需为此增加本 Issue 未规定的改善门槛或声称统计显著性。

    已执行:通过连接器核实 Issue/仓库身份、读取上述方案与固定源码,发表前核对本人无既有评论。未执行:仓库代码、测试、真实 Agent/NPU 运行、原始 campaign archive 哈希验证或硬件资格验证;本意见不宣称这些通过,也不触发设备实验。待证据齐全后,按上述冻结范围做验收增量核查。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:harnessHarness module: profile, A/B loop, quality gates, evidence reportskind:acceptanceCampaign acceptance issue: frozen gates + evidence tree; PASS/FAIL/INCONCLUSIVE

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions