Skip to content

feat: benchmark RSI knowledge with paired optimization runs - #501

Open
Bounty-hunter wants to merge 4 commits into
InferMatrixLab:mainfrom
Bounty-hunter:codex/rsi-knowledge-benchmark
Open

Bounty-hunter wants to merge 4 commits into
InferMatrixLab:mainfrom
Bounty-hunter:codex/rsi-knowledge-benchmark

Conversation

@Bounty-hunter

Copy link
Copy Markdown
Contributor

Refs #337

目前单个 RSI 优化会话可以执行搜索和复测,但要回答“打开知识库是否提高优化效率”,用户仍需手工配置开关组、回收会话和比对结果。这个 PR 新增按框架组件组织的对照入口,首个组件是 Knowledge。同一任务按相同评测协议、预算和 Agent 设置比较开启与关闭知识库时的结果。

用户怎么跑

在支持创建和观察独立 Agent 会话的宿主中,先配置可用的执行环境和设备分配命令,再让 Agent 执行:

inferrsi benchmarks compare --focus knowledge --config benchmark-profile.json --output outputs/knowledge-check

配置也可放在 .inferrsi/benchmarks.json。Knowledge 默认选择 Qwen 去噪与 MoE Forward,每题各跑 on/off 两组,每组 8 轮优化 + 独立 final。可用 --task 选择题目,--rounds 1 做流程快检,或用 --repeats 重复配对。设备充足时,宿主可以用 --parallel 并行派发同题的两组。入口返回实验目录、冻结计划和下一步操作,发起 Agent 按返回信息调用宿主创建与观察会话。

用户拿到什么

compare --action report --output outputs/knowledge-check 输出 离线 HTML、Markdown 和机器可读 JSON。报告先给总体和分任务结论,再展示:

  • 两组的独立 final 加速比、优化耗时、会话 token 及可比较的试验数。
  • 每轮策略、假设、引用的知识、性能和正确性,以及候选是否被采纳。
  • 独立复测的候选身份、校验结果和分用例延迟中位数。折叠区域提供用量与证据,JSON 保留原始采样。
  • 结论可为效率提高、无收益、退化、存在权衡或证据不足。结论适用于配置中选定的任务与重复试验。

改动和执行流程

  • Apps 解析配置,冻结同一计划下的 on/off 组,协调宿主会话交接、回收和报告。
  • Benchmark 定义配对与证据要求,并把 MoE 接到原有候选评测与独立复测入口。
  • Harness 复用单会话的预算、候选留存和 final,额外记录逐轮策略和实际交付的知识引用。
flowchart LR
    A[选择组件、题目与预算] --> B[冻结配对计划]
    B --> C[宿主派发独立 on/off 会话]
    C --> D[逐轮优化并独立 final]
    D --> E[核验候选绑定与配对条件]
    E --> F[输出结论、表格和逐轮证据]
Loading

on/off 使用各自工作区和固定运行时。关闭组移除知识正文并禁用会话检索。其余知识入口由执行指令约束。完整配置及宿主交接见 benchmark comparison。

验证

  • PR 提交通过本地自审、Ruff 和相关回归:188 项通过、2 项跳过、43 项子检查通过。覆盖对照命令、报告、MoE 协议及模块边界。
  • 前一开发版本已在 Ascend 910B1 上完成两题各 on/off 的一轮流程验证。正式 8 轮对照将在本 PR 提交上执行,结果回贴本 PR。

Signed-off-by: dengyunyang <584797741@qq.com>
Agent-assisted-by: Codex (implementation and self-review)
Agent-assisted-by: Codex comparison_rules_report (implementation)
Agent-assisted-by: Codex moe_protocol (implementation)

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent initial review of base cf671a8 → head 9772e69. No previous review from this account on this PR.

P2 — malformed rank evidence can still become a resumable candidate rejection.
In src/inferrsi/benchmark/tasks/moe-forward-npu/optimization.py, adjudicate sets status=candidate_failure when either rank has a candidate* phase/failure_kind, then raises on a malformed phase/failure_kind. The exception preserves that candidate_failure status. In assess (around lines 125–133), the rejected branch only requires two rank_status entries and no baseline_* string; it does not reject this malformed-evidence case.

Concrete source-path example: use the existing rejection fixture (both valid integer rank IDs, normal inner failure and verified cleanup), change rank 1 phase to null while rank 0 remains candidate_import, and recompute the result through _adjudicate. rank_status still has two entries; the error is "Malformed rank phase or failure_kind"; assess matches the recomputed result and returns candidate_failure / FAIL, permitting another optimization round, instead of infrastructure_failure / INCONCLUSIVE for incomplete evidence. The same issue applies to a non-string failure_kind. This is static path reasoning, not an executed reproduction.

Keep round_consumed=true to avoid a free retry, but separate that accounting fact from permission to resume. Require validated rank rejection provenance before returning resumable candidate_failure; malformed/unknown peer evidence must stay blocked. Add negative tests for null/non-string phase and failure_kind alongside the current missing/invalid-rank tests. Merely checking that result.json agrees with the same adjudicator does not provide that missing validity gate.

Other inspected areas

  • Apps separates per-trial workspaces/runtime corpus, freezes settings and manifests, re-verifies evidence when reporting, and compares observed pair settings. Host observations remain caller-exported, not authenticated proof; the docs correctly distinguish the instruction boundary from OS isolation.
  • Comparison rules retain missing/failed pairs in the denominator, compare paired log-ratio medians per task, and do not pool unrelated task latencies. Optional token gaps do not silently become zero.
  • Report rendering escapes narrative and keeps missing values; Harness additions constrain citations to already delivered contexts. These source observations are not execution/visual-QA results.
  • Read the 20-file PR diff, relevant full implementation/test paths, AGENTS changes and governing #337 requirements. The historical device work described by the author is not verification of this head.

Checks and approval prerequisites

  • Current exact-head checks: full-check 3.10/3.11 and dashboard-browser running, macOS queued; Windows, installed-package 3.10/3.11, references and declarations report success. Package-job raw checkout evidence identifies synthetic merge 70e2fd3 (9772e69 into cf671a8); I have not verified complete test-result logs for those jobs.
  • Fix the malformed-rank resumability defect and obtain current-head regression/full-check evidence before approval.
  • Keep the new MoE iterative protocol and paired actual Agent/NPU experiment acceptance separate from CPU fixture success. #337 retains incomplete real-device acceptance, and this PR says the full current-head 8-round experiment is still forthcoming. No performance/knowledge-efficiency PASS is established here.
  • Non-blocking: explicitly document and test interpreter equivalence for collected remote trials. The frozen config contains python, while collection compares environment observations rather than that literal campaign path; differing paths may represent the same interpreter, so a semantic identity rule is preferable to an unexplained path-only assumption.

Actually performed: connector-based source/diff/test-definition review, PR/author/head and own-review checks, check-status lookup and package-log checkout inspection. No repository code, tests, devices or provider calls were executed; no other reviewers' opinions were read; no source changes, merges or auto-merge were performed. This is COMMENT, not approval.

@Dong1017 Dong1017 added kind:task Bounded delivery task with a delegated slice area:harness Harness module: profile, A/B loop, quality gates, evidence reports labels Nov 18, 2026

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR review verdict — DO NOT PASS

Reviewed base cf671a8 and head 9772e69. This is a first review, so the diff is the full PR diff. The diff was truncated in my workspace inside tests/test_moe_optimization_protocol.py, so I did not see the rest of that file.

Prior reviews: there is no formal APPROVED review on this head. The only formal review is @zengchuang-hw's COMMENTED review on this head, and there is no outstanding CHANGES_REQUESTED. That review raises a P2 finding that I re-traced against the diff. It is still open and I treat it as a blocker.

Blocker — malformed rank evidence can become a resumable candidate_failure
File: src/inferrsi/benchmark/tasks/moe-forward-npu/optimization.py, in _adjudicate and in the rejected branch of assess.

Trace:

  1. _adjudicate sets status="candidate_failure" and round_consumed=True when either rank has a candidate_* phase or failure_kind.
  2. It then raises ValueError("Malformed rank phase or failure_kind") when the other rank's phase or failure_kind is not a string, for example null.
  3. The except block only stores result["error"], so the status stays candidate_failure.
  4. result["rank_status"] is built with r.get(k), so it still has two entries.
  5. In assess, recomputed equals the stored result. The rejected branch only checks two rank_status entries and no baseline_* string.
  6. assess therefore returns candidate_failure / FAIL with round_consumed=True. The Harness treats that as a normal failed round and allows the search to continue.
  7. Rank evidence that is incomplete or malformed should stay infrastructure_failure / INCONCLUSIVE, which the diff says blocks later search. The same applies to a non-string failure_kind.

This is static path reasoning. I did not execute it.

The PR makes this path load-bearing. The logic existed in benchmark.py for single-candidate runs. Here it moves into the shared protocol and is reused by the new session and compare flows. There it decides whether an Agent may keep optimizing and whether an on/off trial counts as an honest failed round. A silently resumable round on corrupt evidence is the kind of leak that weakens the comparison.

Requested fix:

  • Keep round_consumed=True so there is no free retry. Separate that accounting fact from permission to resume.
  • Return resumable candidate_failure only when both rank records pass validation. That means each phase and failure_kind is a string, and the candidate-failure provenance comes from validated rank records.
  • Otherwise return incomplete(...), i.e. infrastructure_failure / INCONCLUSIVE.
  • Add negative tests in tests/test_moe_optimization_protocol.py, based on the existing rejection fixture:
    • rank 1 phase=None while rank 0 is candidate_import
    • a non-string failure_kind
    • each must yield infrastructure_failure, not candidate_failure

What I checked and found consistent

  • benchmark.py now imports TASK_FILES, _hashes, _positive, _adjudicate and LAUNCHER_SOURCE from optimization.py. TASK_FILES now includes optimization.py, and _binding hashes it.
  • evaluate_round validates strategy and knowledge_refs. Citations must be unique and must come from contexts already delivered in that campaign. test_round_cannot_cite_undelivered_knowledge covers this.
  • benchmark/comparison.py keeps failed, missing and ineligible trials in the denominator. It uses reciprocal neutral bands and overflow-safe medians, and states that the results are descriptive only.
  • benchmark_compare.py freezes the plan digest and verifies the runtime manifest on every _load. report() re-verifies campaign evidence instead of trusting a stale collection.
  • The off condition omits knowledge/data and flags any delivered context in off.
  • The docs say this is not an OS sandbox and that the MoE iterative protocol is not yet validated on NPU, which is an honest limit.

Not verified here

  • I did not run any tests or CI.
  • I did not see the truncated tail of the MoE protocol test file or any other files after the truncation point.
  • I did not check any NPU execution.
  • I did not check wheel resource packaging for the new optimization.py.

I will re-read after the author pushes a fix. This COMMENT does not approve the change or alter its scope.


Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same-head evidence increment: 9772e69 → same head (no source change). Previous independent review: #501 (review)

Newly verified blocking check failure: both current Python 3.10 and 3.11 full-check jobs fail test_active_consumers_use_canonical_paths_and_are_registered in tests/test_apps_module_migrations.py:108. I read the original logs and independently checked the current source/manifest: benchmark_compare.py imports .storage.archive._publish_directory, but docs/apps-module-migrations.json's inferrsi.apps.storage.archive consumers list does not register src/inferrsi/apps/benchmark_compare.py. Add the actual new consumer to that canonical migration inventory and re-run the existing guard; do not weaken the guard or revert the import to the legacy shim.

Raw evidence:

The previously reported malformed-rank resumability P2 remains unresolved because the head is unchanged. Approval still requires that fix/regression evidence plus the canonical-consumer registration and successful necessary checks. Real Agent/NPU comparison acceptance remains separate from CPU fixtures.

Actually performed: current author/head and own-review dedup check, raw full-check log inspection, direct import/manifest comparison. No repository code or tests executed locally, no other reviewers' opinions read, no merge or branch changes. No new non-blocking suggestion beyond the prior review.

@david6666666 david6666666 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One pair-validity gap in the comparison binder.

if args.action == "bind":
if not args.host or not args.session_id or args.receipt is None:
raise ValueError("bind requires --host, --session-id and --receipt")
return agent_sessions.bind(job, host=args.host, session_id=args.session_id,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This binds the native ID only within one job. The on and off trials can therefore bind the same host/session ID and still become an eligible pair, reusing one conversation and potentially counting its session-total tokens twice. Could we enforce one native session per trial under .compare.lock and mark existing duplicate bindings ineligible during report verification? Same-trial retries can stay idempotent.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d432c03. Compare now rejects a native session ID already bound to another trial on the same host. The check runs under the comparison lock, and retrying a bind for the same trial is idempotent. Report verification marks all trials sharing an existing binding ineligible. Regression tests cover both the bind path and reporting from older job records.

Signed-off-by: dengyunyang <584797741@qq.com>
Agent-assisted-by: Codex (implementation and validation)

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent incremental review: 9772e69 → e2f4a15. Previous review: #501 (review)

Resolved in source: the sole changed line registers src/inferrsi/apps/benchmark_compare.py as a consumer of inferrsi.apps.storage.archive in docs/apps-module-migrations.json. This matches its actual _publish_directory import and the registration guard. The prior missing-consumer failure is therefore addressed statically; I have not claimed a fresh full-suite pass.

Still blocking (P2): malformed rank evidence is accepted as a candidate rejection. benchmark/tasks/moe-forward-npu/optimization.py:262–270 sets candidate_failure as soon as one rank indicates a candidate failure, then validates peer phase/failure_kind types. If the peer phase is malformed, the exception handler at 288–289 records error without restoring infrastructure_failure. With two valid rank identities, assess's len(rank_status)==2 check still passes, and the rejected branch returns FAIL rather than INCONCLUSIVE.

Concrete unexecuted static negative case: retain rank 0 phase="candidate_import" in the rejection fixture and change rank 1 phase to []; recompute the adjudication receipt and assess it with the failed/2 outer result. The code retains a malformed-evidence error but still treats the result as resumable candidate rejection. The new commit does not change this code or its tests.

Before approval: preserve round_consumed=True for already-spent work, but classify invalid peer phase/failure_kind as infrastructure_failure / INCONCLUSIVE; add regression coverage for valid candidate failure mixed with malformed peer evidence. Also obtain necessary fresh-head full-check evidence. At inspection only declarations and a knowledge-model-policy status described as "Not applicable" were present for the new SHA; neither proves the full regression suite passed.

No new non-blocking suggestion in this delta. Independently read AGENTS, actual delta, relevant implementation and test definitions, and current head/check metadata; did not read other reviewers' opinions. No repository code/tests/builds were executed, no branch modified, and nothing merged.

Agent-assisted-by: Codex (integration and validation)
Signed-off-by: dengyunyang <584797741@qq.com>

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent incremental review of e2f4a15 → 264f8ee. Previous review: #501 (review)

This head is a two-parent merge of the previous PR head and main@965ab5cf83399a99530949b04bb934a892ee417c. I separated imported main policy/handoff changes from the PR's benchmark delta; merging main is not itself a fix for the outstanding assessment defect.

Resolved item remains resolved: docs/apps-module-migrations.json still includes src/inferrsi/apps/benchmark_compare.py under the canonical storage.archive consumer list after the handoff migration. I checked the retained _publish_directory import against that exact registration.

P2 remains blocking: src/inferrsi/benchmark/tasks/moe-forward-npu/optimization.py:262–270 still sets candidate_failure before validating the peer's phase/failure_kind types; the exception handler at 288–289 only sets error. Thus a valid candidate failure on one rank plus malformed phase/failure_kind on another valid rank identity can retain candidate_failure and be accepted by assess as FAIL instead of infrastructure_failure / INCONCLUSIVE. The merge does not change this implementation or add the needed negative control.

Required before approval: separate observed round consumption from validity of the failure classification. Preserve round_consumed=True but classify malformed peer evidence as infrastructure failure / INCONCLUSIVE, with a regression mixing a valid candidate rejection and e.g. peer phase=[] or failure_kind=None. Re-run necessary exact-head tests.

Checks at inspection: installed-package 3.10/3.11, references and declarations succeeded; both full-check jobs and host/browser checks are not all complete. knowledge-model-policy reports success with description "Not applicable", not code-validation evidence. Old-head test results cannot establish this integration head's full pass. No new non-blocking suggestion.

Actually inspected: merge parents, both comparison ranges, current AGENTS, actual manifest/import binding, adjudication and assessment source/test context, and exact-head check metadata. No repository code/tests/builds executed, no other reviewers' opinions read, no merge/auto-merge performed.

Agent-assisted-by: Codex (review fixes and regression tests)
Signed-off-by: dengyunyang <584797741@qq.com>
@Bounty-hunter

Copy link
Copy Markdown
Contributor Author

@zengchuang-hw @Dong1017 I fixed the MoE rank-evidence issue in d432c03. Rank metadata is validated before a candidate rejection can be classified. Missing or invalid phase metadata and non-string failure kinds now produce an infrastructure failure / INCONCLUSIVE while retaining the consumed-round flag when candidate work occurred. The regression cases cover these alongside missing and mismatched rank records.

The archive consumer registration is in e2f4a15 and was carried forward when syncing main. I also documented the observed environment fingerprint used to compare remote interpreters and added coverage for equivalent paths and different fingerprints. The targeted suite on the updated branch passed 212 tests and 43 subtests, with 2 skipped. Ruff passed. Full CI and the pinned eight-round NPU comparison are tracked separately. Acceptance issue

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent incremental review: 264f8ee → d432c03. Previous review: #501 (review)

The previously reported MoE P2 is resolved in source. _adjudicate now validates phase presence/nonblank string and failure_kind string before assigning candidate_failure. Candidate work already observed still sets round_consumed=True; malformed peer evidence therefore retains infrastructure_failure instead of establishing a resumable rejection. The regression extends a real rejection fixture with missing/null phase or invalid failure_kind, verifies the adjudication error/status, and requires infrastructure_failure / INCONCLUSIVE with spent round and no score. These are inspected test definitions, not a local execution claim. The earlier migration-consumer registration fix is unchanged.

I independently reviewed all five changed files and adjacent control flow:

  • CLI binding checks every existing trial under the comparison lock, using (host, session_id); optional host_id cannot disguise reuse. Rebinding the same trial remains delegated to the existing idempotent binding validation.
  • Report generation independently recomputes all trial evidence, groups bindings across all repeats/tasks, and excludes every member of a duplicate group. This also catches legacy/direct-binding paths that bypass the new CLI guard.
  • New controls cover same-trial idempotency, cross-trial reuse with changed host_id, legacy duplicate bindings, differing observed environment fingerprints, and equivalent interpreter paths. Documentation now explicitly requires distinct sessions and puts observed Python implementation/version into the environment fingerprint instead of equating executable path spelling with runtime identity.
    No definite new blocking code defect found in this incremental review.

Before approval: successful necessary fresh-head regression and host/installed checks, with any failures or relevant skips inspected. At inspection declarations and pr-references succeeded; full-check 3.10/3.11 and other runtime checks were still running or queued. Older-head results and the author's local claims are not substituted for this head's completed evidence. COMMENT records the resolved source defect while that execution-evidence condition is open.

No additional non-blocking suggestion in this round. Approval of scaffold changes would not establish real knowledge-efficiency gains or NPU campaign acceptance. Rechecked author/repository, exact head and my prior results before submission. No repository code, tests, builds or device work executed by me; no other reviewers' opinions read; no merge or auto-merge.

@david6666666 david6666666 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One remaining rank-evidence validation case.

if any(not isinstance(r.get("phase"), str) or not r["phase"].strip()
or not isinstance(r.get("failure_kind", ""), str) for r in ranks):
raise ValueError("Malformed rank phase or failure_kind")
failed = any(r["phase"].startswith("candidate_")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This now rejects blank and wrong-type metadata, but an unknown nonempty phase still passes validation. For example, a valid candidate_import rejection on rank 0 plus an unrecognized phase on rank 1 becomes a resumable candidate_failure, even though the frozen grader cannot emit that peer phase. Could we validate both ranks against the grader’s phase/failure-kind vocabulary before classifying a rejection? A regression with a valid rejection and an unknown peer phase should return INCONCLUSIVE while preserving round_consumed=True.

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR review verdict — DO NOT PASS

Reviewed base 965ab5c and head d432c03, in response to the fix summary on this head. I had no diff in my workspace. This verdict rests on the PR body, the thread, and the formal and inline reviews, so I did not re-trace the source and I ran no tests.

Review state: No formal APPROVED review exists on this head. Bounty-hunter, zengchuang-hw and david6666666 each have a COMMENTED review on it. There is no outstanding CHANGES_REQUESTED.

Resolved on this head

  • The missing consumer registration for benchmark_compare.py in docs/apps-module-migrations.json was fixed in e2f4a15. zengchuang-hw re-checked it on 264f8ee and again on this head.
  • The duplicate native-session binding is fixed. Compare now rejects a session ID already bound to another trial on the same host, under the comparison lock, and a retry for the same trial stays idempotent. Report verification excludes trials that share an existing binding. zengchuang-hw's review of this head covers it, including legacy duplicate bindings and a changed host_id.
  • The earlier malformed-rank P2 is fixed for blank and wrong-type phase and failure_kind. zengchuang-hw confirmed this in source and in the regression fixtures.

Blocker — open on this head

  • src/inferrsi/benchmark/tasks/moe-forward-npu/optimization.py:266 (david6666666, review on d432c03 plus inline comment). Validation checks only presence and type. A nonempty but unknown peer phase still passes. A valid candidate_import rejection on rank 0 plus an unrecognized phase on rank 1 therefore still becomes a resumable candidate_failure/FAIL. The frozen grader cannot emit that phase.
  • Nothing in the thread says this was addressed, and your fix summary doesn't mention it.
  • Required: validate both ranks' phase and failure_kind against the grader's vocabulary before classifying a rejection. Keep round_consumed=True, but return infrastructure_failure / INCONCLUSIVE for unknown values. Add a regression that pairs a valid rejection with an unknown nonempty peer phase and an unknown failure_kind.

Not yet evidenced (needed before approval)

  • The 212 tests passed / 43 subtests passed / 2 skipped figure and the Ruff pass are your own report. I haven't seen full CI on d432c03. The earlier full-check jobs on 9772e69 failed on the consumer registration. Please confirm the Python 3.10 and 3.11 jobs are green on this head.
  • The pinned 8-round NPU comparison is still pending, as the PR body says. Please post it to the PR or to #502 as acceptance evidence.

I will re-read after the next push.


Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:harness Harness module: profile, A/B loop, quality gates, evidence reports kind:task Bounded delivery task with a delegated slice

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants