Repository navigation
feat: benchmark RSI knowledge with paired optimization runs - #501
Bounty-hunter wants to merge 4 commits into
Conversation
Signed-off-by: dengyunyang <584797741@qq.com> Agent-assisted-by: Codex (implementation and self-review) Agent-assisted-by: Codex comparison_rules_report (implementation) Agent-assisted-by: Codex moe_protocol (implementation)
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent initial review of base cf671a8 → head 9772e69. No previous review from this account on this PR.
P2 — malformed rank evidence can still become a resumable candidate rejection.
In src/inferrsi/benchmark/tasks/moe-forward-npu/optimization.py, adjudicate sets status=candidate_failure when either rank has a candidate* phase/failure_kind, then raises on a malformed phase/failure_kind. The exception preserves that candidate_failure status. In assess (around lines 125–133), the rejected branch only requires two rank_status entries and no baseline_* string; it does not reject this malformed-evidence case.
Concrete source-path example: use the existing rejection fixture (both valid integer rank IDs, normal inner failure and verified cleanup), change rank 1 phase to null while rank 0 remains candidate_import, and recompute the result through _adjudicate. rank_status still has two entries; the error is "Malformed rank phase or failure_kind"; assess matches the recomputed result and returns candidate_failure / FAIL, permitting another optimization round, instead of infrastructure_failure / INCONCLUSIVE for incomplete evidence. The same issue applies to a non-string failure_kind. This is static path reasoning, not an executed reproduction.
Keep round_consumed=true to avoid a free retry, but separate that accounting fact from permission to resume. Require validated rank rejection provenance before returning resumable candidate_failure; malformed/unknown peer evidence must stay blocked. Add negative tests for null/non-string phase and failure_kind alongside the current missing/invalid-rank tests. Merely checking that result.json agrees with the same adjudicator does not provide that missing validity gate.
Other inspected areas
- Apps separates per-trial workspaces/runtime corpus, freezes settings and manifests, re-verifies evidence when reporting, and compares observed pair settings. Host observations remain caller-exported, not authenticated proof; the docs correctly distinguish the instruction boundary from OS isolation.
- Comparison rules retain missing/failed pairs in the denominator, compare paired log-ratio medians per task, and do not pool unrelated task latencies. Optional token gaps do not silently become zero.
- Report rendering escapes narrative and keeps missing values; Harness additions constrain citations to already delivered contexts. These source observations are not execution/visual-QA results.
- Read the 20-file PR diff, relevant full implementation/test paths, AGENTS changes and governing #337 requirements. The historical device work described by the author is not verification of this head.
Checks and approval prerequisites
- Current exact-head checks: full-check 3.10/3.11 and dashboard-browser running, macOS queued; Windows, installed-package 3.10/3.11, references and declarations report success. Package-job raw checkout evidence identifies synthetic merge 70e2fd3 (9772e69 into cf671a8); I have not verified complete test-result logs for those jobs.
- Fix the malformed-rank resumability defect and obtain current-head regression/full-check evidence before approval.
- Keep the new MoE iterative protocol and paired actual Agent/NPU experiment acceptance separate from CPU fixture success. #337 retains incomplete real-device acceptance, and this PR says the full current-head 8-round experiment is still forthcoming. No performance/knowledge-efficiency PASS is established here.
- Non-blocking: explicitly document and test interpreter equivalence for collected remote trials. The frozen config contains python, while collection compares environment observations rather than that literal campaign path; differing paths may represent the same interpreter, so a semantic identity rule is preferable to an unexplained path-only assumption.
Actually performed: connector-based source/diff/test-definition review, PR/author/head and own-review checks, check-status lookup and package-log checkout inspection. No repository code, tests, devices or provider calls were executed; no other reviewers' opinions were read; no source changes, merges or auto-merge were performed. This is COMMENT, not approval.
Dong1017
left a comment
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Reviewed base cf671a8 and head 9772e69. This is a first review, so the diff is the full PR diff. The diff was truncated in my workspace inside tests/test_moe_optimization_protocol.py, so I did not see the rest of that file.
Prior reviews: there is no formal APPROVED review on this head. The only formal review is @zengchuang-hw's COMMENTED review on this head, and there is no outstanding CHANGES_REQUESTED. That review raises a P2 finding that I re-traced against the diff. It is still open and I treat it as a blocker.
Blocker — malformed rank evidence can become a resumable candidate_failure
File: src/inferrsi/benchmark/tasks/moe-forward-npu/optimization.py, in _adjudicate and in the rejected branch of assess.
Trace:
_adjudicatesetsstatus="candidate_failure"andround_consumed=Truewhen either rank has acandidate_*phase orfailure_kind.- It then raises
ValueError("Malformed rank phase or failure_kind")when the other rank'sphaseorfailure_kindis not a string, for examplenull. - The
exceptblock only storesresult["error"], so the status stayscandidate_failure. result["rank_status"]is built withr.get(k), so it still has two entries.- In
assess,recomputedequals the stored result. The rejected branch only checks tworank_statusentries and nobaseline_*string. assesstherefore returnscandidate_failure / FAILwithround_consumed=True. The Harness treats that as a normal failed round and allows the search to continue.- Rank evidence that is incomplete or malformed should stay
infrastructure_failure / INCONCLUSIVE, which the diff says blocks later search. The same applies to a non-stringfailure_kind.
This is static path reasoning. I did not execute it.
The PR makes this path load-bearing. The logic existed in benchmark.py for single-candidate runs. Here it moves into the shared protocol and is reused by the new session and compare flows. There it decides whether an Agent may keep optimizing and whether an on/off trial counts as an honest failed round. A silently resumable round on corrupt evidence is the kind of leak that weakens the comparison.
Requested fix:
- Keep
round_consumed=Trueso there is no free retry. Separate that accounting fact from permission to resume. - Return resumable
candidate_failureonly when both rank records pass validation. That means each phase andfailure_kindis a string, and the candidate-failure provenance comes from validated rank records. - Otherwise return
incomplete(...), i.e.infrastructure_failure / INCONCLUSIVE. - Add negative tests in
tests/test_moe_optimization_protocol.py, based on the existingrejectionfixture:- rank 1
phase=Nonewhile rank 0 iscandidate_import - a non-string
failure_kind - each must yield
infrastructure_failure, notcandidate_failure
- rank 1
What I checked and found consistent
benchmark.pynow importsTASK_FILES,_hashes,_positive,_adjudicateandLAUNCHER_SOURCEfromoptimization.py.TASK_FILESnow includesoptimization.py, and_bindinghashes it.evaluate_roundvalidatesstrategyandknowledge_refs. Citations must be unique and must come from contexts already delivered in that campaign.test_round_cannot_cite_undelivered_knowledgecovers this.benchmark/comparison.pykeeps failed, missing and ineligible trials in the denominator. It uses reciprocal neutral bands and overflow-safe medians, and states that the results are descriptive only.benchmark_compare.pyfreezes the plan digest and verifies the runtime manifest on every_load.report()re-verifies campaign evidence instead of trusting a stale collection.- The off condition omits
knowledge/dataand flags any delivered context in off. - The docs say this is not an OS sandbox and that the MoE iterative protocol is not yet validated on NPU, which is an honest limit.
Not verified here
- I did not run any tests or CI.
- I did not see the truncated tail of the MoE protocol test file or any other files after the truncation point.
- I did not check any NPU execution.
- I did not check wheel resource packaging for the new
optimization.py.
I will re-read after the author pushes a fix. This COMMENT does not approve the change or alter its scope.
Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).
zengchuang-hw
left a comment
There was a problem hiding this comment.
Same-head evidence increment: 9772e69 → same head (no source change). Previous independent review: #501 (review)
Newly verified blocking check failure: both current Python 3.10 and 3.11 full-check jobs fail test_active_consumers_use_canonical_paths_and_are_registered in tests/test_apps_module_migrations.py:108. I read the original logs and independently checked the current source/manifest: benchmark_compare.py imports .storage.archive._publish_directory, but docs/apps-module-migrations.json's inferrsi.apps.storage.archive consumers list does not register src/inferrsi/apps/benchmark_compare.py. Add the actual new consumer to that canonical migration inventory and re-run the existing guard; do not weaken the guard or revert the import to the legacy shim.
Raw evidence:
- Python 3.10: 1 failed, 5263 passed, 37 skipped, 672 subtests passed, 1 warning; 428.96s. https://github.com/InferMatrixLab/InferRSI/actions/runs/44326782850/job/134880096264
- Python 3.11: same counts; 532.38s. Job https://github.com/InferMatrixLab/InferRSI/actions/runs/44326782850/job/134880096606
Both logs explicitly identify synthetic merge 70e2fd3 of this head into cf671a8. These are independently read CI failures, not a conclusion borrowed from another review.
The previously reported malformed-rank resumability P2 remains unresolved because the head is unchanged. Approval still requires that fix/regression evidence plus the canonical-consumer registration and successful necessary checks. Real Agent/NPU comparison acceptance remains separate from CPU fixtures.
Actually performed: current author/head and own-review dedup check, raw full-check log inspection, direct import/manifest comparison. No repository code or tests executed locally, no other reviewers' opinions read, no merge or branch changes. No new non-blocking suggestion beyond the prior review.
david6666666
left a comment
There was a problem hiding this comment.
One pair-validity gap in the comparison binder.
| if args.action == "bind": | ||
| if not args.host or not args.session_id or args.receipt is None: | ||
| raise ValueError("bind requires --host, --session-id and --receipt") | ||
| return agent_sessions.bind(job, host=args.host, session_id=args.session_id, |
There was a problem hiding this comment.
This binds the native ID only within one job. The on and off trials can therefore bind the same host/session ID and still become an eligible pair, reusing one conversation and potentially counting its session-total tokens twice. Could we enforce one native session per trial under .compare.lock and mark existing duplicate bindings ineligible during report verification? Same-trial retries can stay idempotent.
There was a problem hiding this comment.
Fixed in d432c03. Compare now rejects a native session ID already bound to another trial on the same host. The check runs under the comparison lock, and retrying a bind for the same trial is idempotent. Report verification marks all trials sharing an existing binding ineligible. Regression tests cover both the bind path and reporting from older job records.
Signed-off-by: dengyunyang <584797741@qq.com> Agent-assisted-by: Codex (implementation and validation)
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent incremental review: 9772e69 → e2f4a15. Previous review: #501 (review)
Resolved in source: the sole changed line registers src/inferrsi/apps/benchmark_compare.py as a consumer of inferrsi.apps.storage.archive in docs/apps-module-migrations.json. This matches its actual _publish_directory import and the registration guard. The prior missing-consumer failure is therefore addressed statically; I have not claimed a fresh full-suite pass.
Still blocking (P2): malformed rank evidence is accepted as a candidate rejection. benchmark/tasks/moe-forward-npu/optimization.py:262–270 sets candidate_failure as soon as one rank indicates a candidate failure, then validates peer phase/failure_kind types. If the peer phase is malformed, the exception handler at 288–289 records error without restoring infrastructure_failure. With two valid rank identities, assess's len(rank_status)==2 check still passes, and the rejected branch returns FAIL rather than INCONCLUSIVE.
Concrete unexecuted static negative case: retain rank 0 phase="candidate_import" in the rejection fixture and change rank 1 phase to []; recompute the adjudication receipt and assess it with the failed/2 outer result. The code retains a malformed-evidence error but still treats the result as resumable candidate rejection. The new commit does not change this code or its tests.
Before approval: preserve round_consumed=True for already-spent work, but classify invalid peer phase/failure_kind as infrastructure_failure / INCONCLUSIVE; add regression coverage for valid candidate failure mixed with malformed peer evidence. Also obtain necessary fresh-head full-check evidence. At inspection only declarations and a knowledge-model-policy status described as "Not applicable" were present for the new SHA; neither proves the full regression suite passed.
No new non-blocking suggestion in this delta. Independently read AGENTS, actual delta, relevant implementation and test definitions, and current head/check metadata; did not read other reviewers' opinions. No repository code/tests/builds were executed, no branch modified, and nothing merged.
Agent-assisted-by: Codex (integration and validation) Signed-off-by: dengyunyang <584797741@qq.com>
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent incremental review of e2f4a15 → 264f8ee. Previous review: #501 (review)
This head is a two-parent merge of the previous PR head and main@965ab5cf83399a99530949b04bb934a892ee417c. I separated imported main policy/handoff changes from the PR's benchmark delta; merging main is not itself a fix for the outstanding assessment defect.
Resolved item remains resolved: docs/apps-module-migrations.json still includes src/inferrsi/apps/benchmark_compare.py under the canonical storage.archive consumer list after the handoff migration. I checked the retained _publish_directory import against that exact registration.
P2 remains blocking: src/inferrsi/benchmark/tasks/moe-forward-npu/optimization.py:262–270 still sets candidate_failure before validating the peer's phase/failure_kind types; the exception handler at 288–289 only sets error. Thus a valid candidate failure on one rank plus malformed phase/failure_kind on another valid rank identity can retain candidate_failure and be accepted by assess as FAIL instead of infrastructure_failure / INCONCLUSIVE. The merge does not change this implementation or add the needed negative control.
Required before approval: separate observed round consumption from validity of the failure classification. Preserve round_consumed=True but classify malformed peer evidence as infrastructure failure / INCONCLUSIVE, with a regression mixing a valid candidate rejection and e.g. peer phase=[] or failure_kind=None. Re-run necessary exact-head tests.
Checks at inspection: installed-package 3.10/3.11, references and declarations succeeded; both full-check jobs and host/browser checks are not all complete. knowledge-model-policy reports success with description "Not applicable", not code-validation evidence. Old-head test results cannot establish this integration head's full pass. No new non-blocking suggestion.
Actually inspected: merge parents, both comparison ranges, current AGENTS, actual manifest/import binding, adjudication and assessment source/test context, and exact-head check metadata. No repository code/tests/builds executed, no other reviewers' opinions read, no merge/auto-merge performed.
Agent-assisted-by: Codex (review fixes and regression tests) Signed-off-by: dengyunyang <584797741@qq.com>
|
@zengchuang-hw @Dong1017 I fixed the MoE rank-evidence issue in d432c03. Rank metadata is validated before a candidate rejection can be classified. Missing or invalid phase metadata and non-string failure kinds now produce an infrastructure failure / INCONCLUSIVE while retaining the consumed-round flag when candidate work occurred. The regression cases cover these alongside missing and mismatched rank records. The archive consumer registration is in e2f4a15 and was carried forward when syncing main. I also documented the observed environment fingerprint used to compare remote interpreters and added coverage for equivalent paths and different fingerprints. The targeted suite on the updated branch passed 212 tests and 43 subtests, with 2 skipped. Ruff passed. Full CI and the pinned eight-round NPU comparison are tracked separately. Acceptance issue |
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent incremental review: 264f8ee → d432c03. Previous review: #501 (review)
The previously reported MoE P2 is resolved in source. _adjudicate now validates phase presence/nonblank string and failure_kind string before assigning candidate_failure. Candidate work already observed still sets round_consumed=True; malformed peer evidence therefore retains infrastructure_failure instead of establishing a resumable rejection. The regression extends a real rejection fixture with missing/null phase or invalid failure_kind, verifies the adjudication error/status, and requires infrastructure_failure / INCONCLUSIVE with spent round and no score. These are inspected test definitions, not a local execution claim. The earlier migration-consumer registration fix is unchanged.
I independently reviewed all five changed files and adjacent control flow:
- CLI binding checks every existing trial under the comparison lock, using (host, session_id); optional host_id cannot disguise reuse. Rebinding the same trial remains delegated to the existing idempotent binding validation.
- Report generation independently recomputes all trial evidence, groups bindings across all repeats/tasks, and excludes every member of a duplicate group. This also catches legacy/direct-binding paths that bypass the new CLI guard.
- New controls cover same-trial idempotency, cross-trial reuse with changed host_id, legacy duplicate bindings, differing observed environment fingerprints, and equivalent interpreter paths. Documentation now explicitly requires distinct sessions and puts observed Python implementation/version into the environment fingerprint instead of equating executable path spelling with runtime identity.
No definite new blocking code defect found in this incremental review.
Before approval: successful necessary fresh-head regression and host/installed checks, with any failures or relevant skips inspected. At inspection declarations and pr-references succeeded; full-check 3.10/3.11 and other runtime checks were still running or queued. Older-head results and the author's local claims are not substituted for this head's completed evidence. COMMENT records the resolved source defect while that execution-evidence condition is open.
No additional non-blocking suggestion in this round. Approval of scaffold changes would not establish real knowledge-efficiency gains or NPU campaign acceptance. Rechecked author/repository, exact head and my prior results before submission. No repository code, tests, builds or device work executed by me; no other reviewers' opinions read; no merge or auto-merge.
david6666666
left a comment
There was a problem hiding this comment.
One remaining rank-evidence validation case.
| if any(not isinstance(r.get("phase"), str) or not r["phase"].strip() | ||
| or not isinstance(r.get("failure_kind", ""), str) for r in ranks): | ||
| raise ValueError("Malformed rank phase or failure_kind") | ||
| failed = any(r["phase"].startswith("candidate_") |
There was a problem hiding this comment.
This now rejects blank and wrong-type metadata, but an unknown nonempty phase still passes validation. For example, a valid candidate_import rejection on rank 0 plus an unrecognized phase on rank 1 becomes a resumable candidate_failure, even though the frozen grader cannot emit that peer phase. Could we validate both ranks against the grader’s phase/failure-kind vocabulary before classifying a rejection? A regression with a valid rejection and an unknown peer phase should return INCONCLUSIVE while preserving round_consumed=True.
Dong1017
left a comment
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Reviewed base 965ab5c and head d432c03, in response to the fix summary on this head. I had no diff in my workspace. This verdict rests on the PR body, the thread, and the formal and inline reviews, so I did not re-trace the source and I ran no tests.
Review state: No formal APPROVED review exists on this head. Bounty-hunter, zengchuang-hw and david6666666 each have a COMMENTED review on it. There is no outstanding CHANGES_REQUESTED.
Resolved on this head
- The missing consumer registration for
benchmark_compare.pyindocs/apps-module-migrations.jsonwas fixed in e2f4a15. zengchuang-hw re-checked it on 264f8ee and again on this head. - The duplicate native-session binding is fixed. Compare now rejects a session ID already bound to another trial on the same host, under the comparison lock, and a retry for the same trial stays idempotent. Report verification excludes trials that share an existing binding. zengchuang-hw's review of this head covers it, including legacy duplicate bindings and a changed
host_id. - The earlier malformed-rank P2 is fixed for blank and wrong-type phase and
failure_kind. zengchuang-hw confirmed this in source and in the regression fixtures.
Blocker — open on this head
src/inferrsi/benchmark/tasks/moe-forward-npu/optimization.py:266(david6666666, review on d432c03 plus inline comment). Validation checks only presence and type. A nonempty but unknown peer phase still passes. A validcandidate_importrejection on rank 0 plus an unrecognized phase on rank 1 therefore still becomes a resumablecandidate_failure/FAIL. The frozen grader cannot emit that phase.- Nothing in the thread says this was addressed, and your fix summary doesn't mention it.
- Required: validate both ranks'
phaseandfailure_kindagainst the grader's vocabulary before classifying a rejection. Keepround_consumed=True, but returninfrastructure_failure/ INCONCLUSIVE for unknown values. Add a regression that pairs a valid rejection with an unknown nonempty peer phase and an unknownfailure_kind.
Not yet evidenced (needed before approval)
- The 212 tests passed / 43 subtests passed / 2 skipped figure and the Ruff pass are your own report. I haven't seen full CI on d432c03. The earlier full-check jobs on 9772e69 failed on the consumer registration. Please confirm the Python 3.10 and 3.11 jobs are green on this head.
- The pinned 8-round NPU comparison is still pending, as the PR body says. Please post it to the PR or to #502 as acceptance evidence.
I will re-read after the next push.
Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).
Refs #337
目前单个 RSI 优化会话可以执行搜索和复测,但要回答“打开知识库是否提高优化效率”,用户仍需手工配置开关组、回收会话和比对结果。这个 PR 新增按框架组件组织的对照入口,首个组件是 Knowledge。同一任务按相同评测协议、预算和 Agent 设置比较开启与关闭知识库时的结果。
用户怎么跑
在支持创建和观察独立 Agent 会话的宿主中,先配置可用的执行环境和设备分配命令,再让 Agent 执行:
配置也可放在
.inferrsi/benchmarks.json。Knowledge 默认选择 Qwen 去噪与 MoE Forward,每题各跑 on/off 两组,每组 8 轮优化 + 独立 final。可用--task选择题目,--rounds 1做流程快检,或用--repeats重复配对。设备充足时,宿主可以用--parallel并行派发同题的两组。入口返回实验目录、冻结计划和下一步操作,发起 Agent 按返回信息调用宿主创建与观察会话。用户拿到什么
compare --action report --output outputs/knowledge-check输出 离线 HTML、Markdown 和机器可读 JSON。报告先给总体和分任务结论,再展示:改动和执行流程
flowchart LR A[选择组件、题目与预算] --> B[冻结配对计划] B --> C[宿主派发独立 on/off 会话] C --> D[逐轮优化并独立 final] D --> E[核验候选绑定与配对条件] E --> F[输出结论、表格和逐轮证据]on/off 使用各自工作区和固定运行时。关闭组移除知识正文并禁用会话检索。其余知识入口由执行指令约束。完整配置及宿主交接见 benchmark comparison。
验证