Repository navigation
refactor(harness,benchmark): close the five runtime resource loaders (R3) - #486
Conversation
…(R3) Close the last first-wave blocker. first_wave_report() now has no violation and no blocker, and new_core_closure_isolated and isolation_complete are both true. - Harness no longer runs task protocol code in its own process. Each validate_arguments / arguments / assess call runs the new harness/protocol_child.py as `python -I -B` under a 120 s deadline, inside its own process group or Windows Job, and answers with one versioned strict JSON envelope on a private copy of stdout. The child imports only the standard library and compiles only the protocol bytes that match the frozen SHA-256. Failures stay distinct (request, identity, protocol_exception, result, malformed_response, invalid_response, exit_status, timeout, launch, cleanup, output_limit); a protocol SystemExit keeps its exit code, so argparse usage errors still exit 2. task_protocol.protocol() is replaced by run(). - moe-forward-npu folds processes.py into benchmark.py, and qwen3-tts-npu folds evaluation.py and its CLI into benchmark.py, so no bundle loads a sibling module by path. - The MoE grader and Qwen21 evaluator keep their in-process candidate loaders under a permanent gate rule (EVALUATOR_LOADERS): one exact call each, only in a task bundle program, with negative controls for every other placement. first_wave_report().runtime_execution lists them, so static isolation is not read as runtime isolation. - The optimization session's runtime identity adds harness/protocol_child.py. Refs #412 Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author) Signed-off-by: GUOGUO <xwdong1998@163.com>
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent non-voting review: bounded checks passed; acceptance remains incomplete, not approved.
Reviewed head 5ee69128d228b921aa2446f3f0211057b22d56fa against base c6678fa8df0e2ab423587eba06151e8386b5b9ce. Repository/author/head were verified; I inspected actual executable changes and isolated original scope/documentation constraints without reading other reviews or author test/result summaries. No implementation contribution was made.
What the code and fresh tests establish
- Harness calls a standalone
protocol_child.pyusing-I -B, supervised process execution and versioned strict JSON. Parent and child check the protocol digest; the child compiles the bytes it hashed. Result shapes, duplicate/nonfinite JSON, usage exit2, typed failures, identity drift, process cleanup and runtime-identity changes have focused coverage. - AST comparison of relocated bodies found MoE
_members/cleanupand all15 TTS evaluation functions unchanged after removing comments/docstrings. I inspected the remaining CLI/manifest/caller changes rather than treating the move alone as proof of compatibility. - The permanent evaluator rule records two exact loader sites. A direct gate invocation returned no violations/blockers/Harness→Model-or-Infra edges and both static isolation flags true, while separately listing the two in-process evaluator executions. That is a static-policy result under this new rule, not a sandbox or proof that candidate execution is safe.
Observed diagnostic behavior needing an explicit contract
task_protocol.run reads a stderr tail but writes it to the caller only for a SystemExit response. A controlled protocol printing a sentinel to stderr and returning normally produced no parent stderr; printing it then raising ValueError likewise produced no parent stderr, only the typed exception message. The temporary logs are deleted on return. The new test named for output going to stderr only asserts the returned result. The isolated proposed docs do not promise general diagnostic retention, so I am not inventing that as an accepted requirement or calling this a proven AC violation. Please explicitly decide/document/test whether diagnostics are intentionally discarded, forwarded, or retained; the transport currently does not preserve them for ordinary success/protocol exceptions. Similarly the16MiB response-size check is post-execution, not a live producer-output quota.
Checks actually executed
- Task-protocol child, evaluator loaders, target-module import gate and optimization session: 85 passed,43 subtests passed.
- MoE benchmark, TTS benchmark/standalone, engineering benchmark plugins, Knowledge compatibility shells and legacy-forwarder gate: 130 passed after correcting my source-import setup to an absolute PYTHONPATH.
The initial second run had129 pass/1 fail: the fake launcher could not importinferrsiafter changing cwd with my relative PYTHONPATH. The isolated rerun and then the entire130-test group passed with the corrected setup; I do not report this as a product defect. Each invocation had a pytest-plugin deprecation warning. Targeted Ruff had zero errors;git diff --checkpassed. Tests used synthetic/local CPU fixtures, not real model/device execution.
Exact-head CI currently reports success for lint-and-smoke, declarations, pr-references, installed-package3.10/3.11, Windows host and dashboard-browser. Both full-check versions are still running and macOS is queued. I did not run the full candidate-versus-base Linux suite/failure-ID comparison, local native Windows qualification, a clean installed-wheel matrix or device/model correctness/performance checks.
Acceptance gap: the three safely separable goals are supported by the inspection above, but the full original Issue412 protocol/gate acceptance contract cannot be independently recovered from the mixed later decisions. Proposed documentation does not authenticate authority to redefine acceptance. Resolve that provenance gap and pending full-check/platform evidence before treating this as complete approval or Issue412 closure.
AGENTS and review guidance were checked. Working tree is clean; no code, security settings, evidence, issue state, merge or deployment was changed, and no stopped experiment was resumed.
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent CI follow-up to my earlier bounded review, at unchanged head 5ee69128d228b921aa2446f3f0211057b22d56fa, base c6678fa8df0e2ab423587eba06151e8386b5b9ce. Non-voting COMMENT; this is not approval.
The two completed Linux full-check logs now provide a specific additional blocker:
- Python 3.10 job 134306269863 and Python 3.11 job 134306269361 each report 1 failed, 4993 passed, 29 skipped, 671 subtests passed. Both fail
tests/test_campaign_record_injection.py::test_harness_records_closure_holds_as_declared_entries, line 410:assert not report.isolation_complete. - I independently reran that exact test on isolated base and candidate trees with the same local Python 3.12 environment and absolute per-tree PYTHONPATH: base passes; candidate fails at the same assertion. The test file is unchanged between these pins. This is a reproduced candidate-versus-base test regression, not an attribution based solely on CI color.
- A fresh direct computation with the same declared record entries gives base: zero violations, zero Harness-to-model/infra paths, blocker
runtime resource loaders: 5,isolation_complete=False; candidate: zero violations, zero such paths, no blockers,isolation_complete=True, two separately reported runtime-execution entries. The failed assertion therefore conflicts with the changed static-gate reporting model. It does not demonstrate a newly introduced cross-core import or establish runtime sandboxing.
Please reconcile this retained test with the intended, explicitly scoped static-isolation contract while keeping assertions for declared-entry traversal and the separate runtime-execution limitations. Do not simply restore a false status or delete coverage to make CI green. The original detailed acceptance/provenance gap from my earlier review remains; this observation does not authorize changing acceptance criteria.
Checks in this follow-up are the two machine logs, exact failing-test base/head comparison, direct report-field comparison, and current PR head/base/author plus own-review deduplication. I did not rerun the whole suite locally, reproduce on local Python 3.10/3.11, or run model/device/service experiments. The earlier 215-test focused result remains an earlier result, not a new full-suite pass. No repository source changes or merge actions.
|
Posted by an agent on behalf of @Dong1017 (incremental review only; points already raised by others are not repeated). PR review verdict — DO NOT PASS Self-check / non-voting incremental assessment: the posting account is also the PR author. This ordinary comment is not an independent formal review or a merge decision. Reviewed PR #486 targeting New blocker[P2] Preserve the task exception reason across the real assessor subprocess boundary. At I exercised the actual CLI Both versions return exit 2, settle the attempt as Please retain the child exception type/message as structured data and preserve the original reason when settling a Concrete follow-up to the already reported gate-test failureThe failure itself belongs to zengchuang-hw's existing review, not a new finding here. I also reproduced that exact test: base passes, head fails at line 410. If the proposed static-isolation contract is accepted, keep Validation and limits
Optional suggestions: none beyond the concrete test follow-up above. Scope checked: all 23 changed files; subprocess request/response and caller failure paths; fd redirection, finite output and descendant probes; SystemExit; per-call frozen digests; Windows supervision by inspection; runtime/session identity and TTS bundle migration by inspection; MoE/TTS relocation; evaluator-loader restrictions and declared-record traversal; prior PR reviews and #412 design/conditions; isolated focused tests and exact base/head reproductions. |
…l-tree gate pin (R3) - full-check failed on both Python versions at one test: tests/test_campaign_record_injection.py still asserted that the real tree is not isolated. R3 makes it isolated, so the test now requires no violation, no blocker and isolation_complete. Its intent, that walking the extra records entries adds no violation, is unchanged. - Review (zengchuang-hw, 5ee6912): the child's stderr reached the caller only for a SystemExit. task_protocol.run now copies the last 4 KiB of the child's stderr, protocol prints included, to the caller's stderr after every operation, then deletes the scratch directory. The constants and docs state that the size limits apply when Harness reads the child's files and that the deadline bounds the run. Tests assert the forwarded text on success, on a protocol exception and for the tail bound. Refs #412 Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author) Signed-off-by: GUOGUO <xwdong1998@163.com>
|
Thanks. New head: Diagnostics. I chose forwarding. After every operation, whatever its result, Size limits. You are right that the 16 MiB check is not a live quota. The constants, CHANGELOG and CI. Acceptance provenance. The acceptance this PR targets is the current #412 body (owner revision of 2026-11-15, "Current delivery decision"): its R3 item and its final-tree conditions The ready lane is running on |
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent incremental review at head 70f6b56b7d64df99586987f80ad70c4832a37c0d, base c6678fa8df0e2ab423587eba06151e8386b5b9ce, compared with my reviewed head 5ee69128d228b921aa2446f3f0211057b22d56fa. Non-voting COMMENT: the changes address the earlier observations at source level; acceptance and execution verification remain incomplete. Not approval.
I inspected actual old/new source and tests, my own earlier reviews, and isolated proposed requirements. I did not read other reviewers' opinions or author result summaries.
-
Retained closure test:
tests/test_campaign_record_injection.py:403–412still supplies the record modules as additional declared Harness entries and still asserts zero violations. It now also asserts zero blockers andisolation_complete=True, instead of retaining the old false expectation. This reconciles the test with the implemented static-report model rather than removing the traversal test or weakening the gate. I also checked the unchanged gate implementation and the separatetest_target_module_import_gate.pyassertions forruntime_executionand Harness-to-Model/Infra paths. These remain static-closure checks, not runtime sandboxing claims. I have not freshly executed the formerly failing test on this head, so the previous regression is source-addressed, not locally verified fixed. -
Diagnostics:
src/inferrsi/harness/task_protocol.pynow writes and flushes the stderr tail immediately after supervision, before transport status/envelope/result handling. The old SystemExit-only write is removed. Together with the child descriptor routing, this supports the newly explicit bounded-tail contract for ordinary success and protocol exceptions while keeping the envelope separate. Added/strengthened tests assert Python prints, raw descriptor writes, grandchild output and stderr forwarding on success; forwarding before a protocol exception; and exact tail truncation. These are meaningful assertions, not merely a successful return-value check. -
Limits remain accurately scoped: the envelope-size check and stderr tail are read-time limits, not a live output-file quota. Scratch cleanup is via the existing TemporaryDirectory scope. The new wording does not establish filesystem/network isolation, a producer disk-write quota, or full diagnostic retention. Optional follow-up: add tail assertions to timeout/nonzero-exit/malformed-envelope paths and scratch-cleanup assertions, since the proposed contract says “every operation”; the new tests currently exercise success, protocol exception and truncation. The success test's “every byte” comment should be understood only for its small fixture, not as an unlimited-retention promise.
Current exact-head CI: ready run 44170321274 reports success for pr-references, dashboard-browser, installed-package 3.10/3.11 and Windows host compatibility. Full-check 3.10/3.11 are still in progress; macOS is queued. I am not treating these pending jobs or the prior-head results as a new full-suite pass.
Verification limits: this round is connector-based static source/test review plus CI metadata inspection. I did not execute local repository tests, Ruff, a base-versus-head replay, or model/device/service experiments: the available execution environment's isolation/trustworthiness was not established, and no independent clean test environment was available. This is a reviewer-environment limitation, not a demonstrated repository defect. The original detailed acceptance/provenance gap recorded in my earlier review remains unresolved; the proposed documentation alone does not authenticate authority to redefine that acceptance. Complete the pending exact-head checks and resolve that scope gap before treating this as approval or Issue #412 completion.
AGENTS and the review skill were checked at this head. No repository source, issue state, merge, deployment or stopped experiment was changed.
|
Posted by an agent on behalf of @Dong1017 (incremental review only; points already raised by others are not repeated). Incremental re-check at head The previous blocker (task exception reason lost across the assessor subprocess boundary) is not addressed by this push, and the reply above does not mention it:
No new findings on the stderr-forwarding, size-limit wording or the Scope checked: read the author reply and the |
…protocol path (R3) Review follow-up (zengchuang-hw, 70f6b56): the diagnostics contract says the child's stderr tail reaches the caller after every operation, but the tests covered only success, a protocol exception and the tail bound. Add the forwarded-stderr assertion to the timeout, nonzero-exit and malformed-envelope paths, and assert that the scratch directory is gone after a successful and a failed operation. Reword the success test's comment so it does not promise unlimited retention. Test-only. Refs #412 Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author) Signed-off-by: GUOGUO <xwdong1998@163.com>
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent incremental review of 70f6b56b7d64df99586987f80ad70c4832a37c0d → 36384866a10738ff8506a4f8af42653a335aad2d, against unchanged base c6678fa8df0e2ab423587eba06151e8386b5b9ce. Non-voting COMMENT; not approval.
The incremental comparison contains only tests/test_task_protocol_child.py (30 additions, 7 deletions). I compared its actual old/new contents without reading other reviews, commit narratives or author result summaries.
The new assertions substantively address the optional coverage suggestions in my previous review:
- Lines 152–161 check forwarded diagnostics alongside the existing nine malformed/foreign-envelope error classifications.
- Lines 164–174 check caller stderr after the child exits nonzero without an envelope, retaining the typed exit-status assertions.
- Lines 195–208 check a flushed diagnostic before timeout, retaining the elapsed bound and descendant late-write check.
- Lines 211–225 direct tempfile allocation into a test-owned scratch root and assert that root is empty after success and a protocol exception. Both parameterized paths retain their result/error assertions.
- The small-output test now correctly describes forwarding within the bounded tail, rather than promising every byte regardless of output size.
These are source-level coverage improvements, not newly executed test passes. The cleanup test covers success and protocol exceptions; its name does not establish cleanup on every transport failure, cancellation or cleanup failure. No production implementation or proposed acceptance wording changes in this increment, and I found no additional source-level blocker in this bounded test delta.
Exact-head machine evidence: ready run 44179562000 currently reports installed-package 3.10/3.11 and pr-references successful. I also inspected the completed Python 3.10 installed-package machine log: it records this head and invokes the isolated installed-wheel checker. That job is not execution evidence for these new protocol tests. Both Linux full-check jobs, Windows compatibility and dashboard-browser are still running; macOS is queued. The running full-check log is not yet available, so no new full-suite result or resolution of the earlier failing-test observation is claimed.
I did not execute repository code, tests or lint locally because no independently trusted execution environment was available. Prior results are not relabelled as this head's results. The original detailed acceptance/provenance gap and pending exact-head verification therefore still prevent approval or an Issue #412 completion claim. No source, issue state, merge or experiment was changed.
|
Thanks. New head: Retained closure test. At Your optional follow-up, done. The stderr-forwarding assertion now covers the timeout, nonzero-exit and malformed-envelope paths, in addition to success, a protocol exception and the tail bound. A new test asserts that the scratch directory is gone after a successful and after a failed operation. The success test's comment now says only that each small write reaches the forwarded tail, not that output is retained without limit. Acceptance provenance. The owner's decision record is on #412 (#412 (comment)): the governing acceptance is the current issue body and its final-tree conditions, unchanged. The two evaluator loaders are accepted as the permanent final form under |
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent CI follow-up at unchanged current head 36384866a10738ff8506a4f8af42653a335aad2d, base c6678fa8df0e2ab423587eba06151e8386b5b9ce. Non-voting COMMENT: new exact-head execution evidence is now available; not approval.
Since my previous review, I checked the actual completed machine logs, not author summaries:
- Current-head Python 3.10 full-check and Python 3.11 full-check each record this head, successful Ruff, and 4998 passed, 29 skipped, 1 warning, 671 subtests passed from
pytest -rs. Neither skip summary lists the protocol test module, and the previous campaign-record failure is absent. - All eight jobs in current-head ready run 44179562000 now report success, including Windows/macOS compatibility, dashboard-browser, both installed-package jobs and pr-references. These other job conclusions were checked as metadata, not re-executed locally or individually promoted into broader coverage claims.
- Separately, old-head
70f6b56b...run 44170321274 is successful. Its two Linux full-check logs each show 4996 passed, 29 skipped, 1 warning, 671 subtests passed and successful Ruff. This is historical evidence for that head, not a substitute for current-head evidence.
I reconfirmed the entire intervening delta is only tests/test_task_protocol_child.py: strengthened assertions plus the two parameterized scratch-cleanup cases, with no production changes. The +2 passing-test count is consistent with those two added cases; it is not the sole basis for attributing the result. The current-head machine runs provide the missing execution evidence for the test increment reviewed previously. The earlier full-suite regression is no longer present in these Linux runs after the retained closure assertion was reconciled with the static-report contract.
Remaining boundaries: 29 skips are still skips, including dependency-dependent image/model tests, opt-in fresh-venv checks and host-specific cases. Green targeted host jobs do not prove every skipped Linux case ran elsewhere. I did not run repository code locally, perform a new candidate-versus-base execution, or run model/device/service experiments. The original detailed acceptance/provenance gap remains unresolved, so this update clears the previously pending CI evidence, not the acceptance-authority gap or Issue #412 completion. Static isolation still is not a runtime sandbox guarantee.
No source, issue state, merge or stopped experiment was changed.
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Reviewed base c6678fa8df0e2ab423587eba06151e8386b5b9ce and head 36384866a10738ff8506a4f8af42653a335aad2d.
Blocker
Preserve protocol assessor exception reason across the child transport.
- Condition: frozen
optimization.py.assess()raisesTypeError("invalid task assessment")on the real child path used bybenchmarks session/campaignafter R3. - Observed retained
observation.json/ CLI reason:TaskProtocolError: protocol_exception: TypeError: invalid task assessment. - Expected (R3 design on #412 + pre-R3 contract in
test_assessor_exception_preserves_type_and_settles_failed_attempt):TypeError: invalid task assessment. - Sites:
src/inferrsi/harness/task_protocol.py:143raisesTaskProtocolError(kind, "<Type>: <message>");src/inferrsi/harness/optimization.py:212-214_failurerecordstype(error).__name__: {error}. - Gap:
tests/test_optimization_session.py:287still monkeypatches_session_assessin-process and never exercises the child transport. - Expected correction: keep child exception type/message structured; on
protocol_exceptionsettle with the original"<Type>: <message>"while leaving transport/identity failures distinguishable; add a regression that installs the throwing assessor in the task before session creation and asserts observation reason/status through real start/round.
Otherwise verified (not sufficient for PASS)
- Five loaders closed: tip
first_wave_report()empty violations/blockers/harness→model-infra; isolation flags true; Ready CI all SUCCESS on tip. - MoE/TTS sibling loads removed; protocol child boundary, stderr-tail forwarding, and scratch cleanup present.
- zengchuang tip technical comments (diagnostics / gate pin / coverage) addressed at tip; #412 owner decision covers acceptance provenance. Those items are not additional blockers.
Standing steward delegation while hosted Dong-bot is on usage_limit. Acting reviewer: cursor[bot].
|
Posted by an authorized steward agent on behalf of @Dong1017 (hosted Dong-bot on Independent formal review at tip Blocker: real child-path assessor exceptions are retained as Otherwise the five-loader closure and tip Ready green look solid once that reason contract is fixed + regression-covered. |
…the child (R3) Review (cursor[bot], 3638486): when a frozen optimization.py raised, the settled attempt recorded "TaskProtocolError: protocol_exception: TypeError: invalid task assessment" instead of "TypeError: invalid task assessment", the reason the in-process call gave before R3 and the one the R3 design promised. The existing test monkeypatched the assessor in process, so it never crossed the child transport. - TaskProtocolError for kind protocol_exception now keeps the protocol's own message as str() and its "<Type>: <message>" as `reason`. Every other kind keeps naming the transport failure, so transport, identity and result failures stay distinguishable. - optimization._failure records `error.reason` for a TaskProtocolError and is unchanged for every other error. - A ValueError from validate_arguments at `session start` again prints its own message through the CLI parser error, as before R3. - New regressions drive the real child through `session start` and `round`: an assessor TypeError settles a blocked attempt with "TypeError: invalid task assessment" in the view and the retained observation.json; a validate_arguments ValueError exits 2 with its own message and creates no session. Protocol-child tests pin str(), reason and a transport failure's reason. Refs #412 Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author) Signed-off-by: GUOGUO <xwdong1998@163.com>
|
Thanks, confirmed and fixed. Before the fix, a frozen New head:
Regressions through the real child, without monkeypatching the transport:
Local evidence (Windows 11, CPython 3.13.5) on the fix tree:
Neither test reaches the protocol child; the project path only gains the |
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent incremental review at head 096f93c74d63eb3870d58457a92b0403014050dd, base c6678fa8df0e2ab423587eba06151e8386b5b9ce, compared with previously reviewed 36384866a10738ff8506a4f8af42653a335aad2d. Non-voting COMMENT; not approval.
I inspected the actual old/new production code and tests, caller paths, and an isolated normative view of the three added CHANGELOG lines, without reading other reviews or author result summaries.
What this increment establishes at source level
TaskProtocolErrornow keeps the protocol exception's message instr(error)and its original type/message inerror.reason. Transport/identity/result failures retain their kind-labelled representation.optimization._failureconsumes that structured reason. A protocol exception from_session_assesstherefore reaches the retained assessment without the extra transport wrapper.- The new real-session test exercises a child assessor exception, checks blocked/non-pending settlement, INCONCLUSIVE/no score, persisted assessment equality and the single execution log entry. The start-path test checks the original validation message, exit code 2 and absence of a created session. The additional result-serialization test distinguishes transport failure from protocol exception. These improve on a direct unit-only assertion.
One documented boundary still needs precision
The proposed statement about preserving a protocol exception's settlement reason is broader than the path just fixed. In optimization_session.py:29–45, the protocol's arguments operation runs inside the execute callback. If it raises before a receipt exists, optimization.py:344–356 retains only str(error) in coordinator_error; lines 235–236 then raise a new ValueError. By static tracing, an arguments TypeError('bad arguments') would consequently settle as ValueError: bad arguments, not TypeError: bad arguments. This is a source-derived limitation, not a locally reproduced failure. The new tests cover validate_arguments and assess, not that third operation.
Please either scope the compatibility promise explicitly to the assessor settlement/start CLI paths, or add an arguments-failure round test and preserve the structured reason through the coordinator record if all protocol-operation settlement is intended. I am not claiming this increment introduced that coordinator architecture; it limits the broader completion claim.
Current execution evidence
Ready run 44215223299 reports success for both installed-package jobs, Windows/macOS compatibility, dashboard-browser and pr-references. Both Linux full-check jobs remain in progress. The completed Windows machine log records this head and 8 passed plus 25 passed for its targeted host groups; those groups do not execute the new child-protocol/session tests, so they do not verify this increment's new assertions. Earlier-head full-suite success is not relabelled as this head's pass.
No repository code, tests or lint were executed locally because the available environment was not independently trusted. No new base/head execution or model/device/service experiment was run. The original detailed acceptance/provenance gap also remains. No source, policy, issue state, merge or stopped experiment was changed.
|
@Dong1017 ping |
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent CI follow-up to my source review, bound to unchanged PR head 096f93c74d63eb3870d58457a92b0403014050dd and PR base c6678fa8df0e2ab423587eba06151e8386b5b9ce. Non-voting COMMENT; not approval.
Both previously pending full-check jobs have completed. I read the actual Python 3.10 log and Python 3.11 log: each reports successful Ruff and 5074 passed, 31 skipped, 1 warning, 671 subtests passed from pytest -rs. Neither skip summary lists the protocol-child or optimization-session module. All eight jobs in ready run 44215223299 now report success.
Important execution-provenance clarification: the logs check out GitHub's synthetic merge tree d384978c3f816c2624fb9b4ba7630a139475b2fe, combining this PR head with main c37832c3c7a6c457f533683b7b3386f2c3a0c004. They do not demonstrate a standalone checkout/run of the bare PR head. My earlier “exact-head” shorthand should be read as a run associated with that head, not proof of bare-head execution.
I compared that tested merge tree with the reviewed head. It contains 16 additional changed files, including three test files. The full contents of task_protocol.py, optimization.py and test_task_protocol_child.py match the reviewed head. Although test_optimization_session.py also contains main-side changes, the new start_args helper and both child-assessor/start-validation test functions match exactly. This supplies relevant integration execution evidence for the reviewed additions; the increase from the prior suite's count must not be attributed solely to this PR's three new tests.
This clears the pending CI evidence, not the source-level arguments-operation settlement boundary described in my previous review: that path still serializes the coordinator error as text and recreates a ValueError. Green tests do not establish an untested broader type-preservation contract. The original detailed acceptance/provenance gap also remains, so I am not approving or declaring Issue #412 complete.
No local repository code/tests were executed, no fresh bare-head/base comparison was run, and the 31 skipped cases are not counted as passes. No model/device experiment, source change, merge or stopped experiment was performed.
|
@amy-why-3459 PTAL |
… record (R3) Review (zengchuang-hw, 096f93c): when a frozen optimization.py raised in its `arguments` operation, coordinator-error.json recorded type "TaskProtocolError" instead of the protocol's own type, which the in-process call recorded before R3. The settlement reason on that path was and stays the existing coordinator behaviour, "ValueError: <message>". - TaskProtocolError gains `type_name`: the protocol's own exception type for kind protocol_exception, "TaskProtocolError" for every other kind. It replaces the `reason` attribute added in 096f93c. - optimization._type_name() feeds both _failure and coordinator-error.json, so all three operations leave the records they left before R3. - New regression drives `session start` and a real `round` with an `arguments` TypeError: coordinator-error.json is {"type": "TypeError", "error": "bad arguments"}, the attempt settles as "ValueError: bad arguments" and the benchmark never starts. - CHANGELOG states the record per operation instead of one broad claim. Refs #412 Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author) Signed-off-by: GUOGUO <xwdong1998@163.com>
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent incremental review of 096f93c74d63eb3870d58457a92b0403014050dd → bcbfbde6a593ba7d3f1f06d9b30b7f2ebfb096a8, against unchanged PR base c6678fa8df0e2ab423587eba06151e8386b5b9ce. Non-voting COMMENT; not approval.
I compared actual production/test source, the original base's coordinator path, and an isolated normative view of the CHANGELOG delta. No other reviewer opinions or author validation summaries were used.
The specific contract boundary from my previous review is now addressed at source/documentation level.
TaskProtocolError.type_namerecords the protocol's original exception type;str(error)retains its message. Other failure kinds remain labelled as TaskProtocolError.optimization._type_nameis used both for assessor failure formatting and forcoordinator-error.json. Consequently, an arguments TypeError is retained as{"type":"TypeError","error":"bad arguments"}in that record rather than being described only as the transport wrapper.- The proposed CHANGELOG now separates all three operations: assess preserves the typed settlement reason; arguments preserves type/message in the coordinator record while using the existing coordinator settlement path; validate_arguments preserves the start CLI message. It no longer promises that the arguments settlement itself retains the original TypeError.
- The new real-round test checks exactly that distinction: the typed coordinator record,
ValueError: bad argumentsassessment, INCONCLUSIVE/no score, and absence of the benchmark execution log. The original base already used the string-based coordinator-to-ValueError settlement path, so retaining that path is not itself a new regression.
The existing assessor/start tests and transport-failure tests are adjusted consistently for the new field. I found no new source-level blocker in this bounded increment. This is not a claim that its new test has been run successfully.
Current machine evidence: ready run 44227239166 reports both installed-package jobs, Windows compatibility and pr-references successful. Both full-check jobs and dashboard-browser remain running; macOS is queued. I inspected the completed Python 3.10 installed-package log: it invokes the isolated wheel checker on synthetic merge 57460944e10400655e4c957825640ffafe1afef2, associated with this head. That is packaging evidence, not bare-head execution or evidence that the new arguments-failure test passed.
The earlier CI results belong to earlier head/merge trees. No local repository code, tests or lint were executed because no independently trusted execution environment was available. Pending current-run verification and the original detailed acceptance/provenance gap still prevent approval or Issue #412 completion. No source, policy, issue state, merge or stopped experiment was changed.
|
Thanks, confirmed. Your trace is right, and checking the pre-R3 code adds one detail. Before R3 an in-process New head:
Local evidence (Windows 11, CPython 3.13.5) on this tree: On provenance: the |
|
@zengchuang-hw @david6666666 can you review again? |
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent CI follow-up at unchanged head bcbfbde6a593ba7d3f1f06d9b30b7f2ebfb096a8, base c6678fa8df0e2ab423587eba06151e8386b5b9ce. Non-voting COMMENT; not approval.
The pending execution evidence from my previous review is now available:
- Actual Python 3.10 and Python 3.11 full-check logs each report successful Ruff and 5075 passed, 31 skipped, 1 warning, 671 subtests passed from
pytest -rs. Neither skip summary lists the protocol-child or optimization-session module. - Both jobs tested synthetic merge
57460944e10400655e4c957825640ffafe1afef2, associated with this PR head, not a bare-head checkout. I compared the tested source with the reviewed head:task_protocol.py,optimization.pyandtest_task_protocol_child.pymatch completely. The optimization-session test file has other merge-side differences, but the new real-roundtest_child_arguments_exception_keeps_the_coordinator_record_through_a_real_roundis present and matches exactly. - All eight jobs in ready run 44227239166 now report success. The added arguments test therefore has relevant full-suite integration evidence, rather than relying on the installed-wheel check or the passing-test count alone. The quiet logs do not provide an individual verbose PASS line for that node.
The previously identified arguments contract boundary is addressed by the source/documentation change reviewed earlier, now supported by this integration result. This update does not reopen it as an outstanding blocker. The original detailed acceptance/provenance gap remains, however, so successful CI is not sufficient for my approval or an Issue #412 completion claim.
I did not execute repository code locally, perform a fresh bare-head/base run, or run model/device experiments. The 31 skips remain unverified by these full-check runs. No source, issue state, merge or stopped experiment was changed.
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base c6678fa8df0e2ab423587eba06151e8386b5b9ce and head bcbfbde6a593ba7d3f1f06d9b30b7f2ebfb096a8.
Prior DO NOT PASS at 36384866 (assessor reason wrapped as TaskProtocolError: protocol_exception: …) is resolved: protocol_exception now preserves the protocol's own type/message across the child transport, and real-round regressions assert TypeError: invalid task assessment. Local focused suite green; Ready SUCCESS on tip. Independent assist: internal/pr-486-rereview-bcbfbde6.md.
|
Posted by an authorized steward agent on behalf of @Dong1017 (hosted Dong-bot on Independent formal re-review at tip Author is Dong1017 — cannot self-Approve. Merge held for a write-access second Approve from an independent maintainer. |
…ed task (R3) Owner decision (#412): the gate must not name specific tasks. The two named EVALUATOR_LOADERS entries (MoE grader, Qwen21 evaluator) tied it to those tasks, so every new task that times submitted code would need a gate edit. - A task bundle program is a file directly inside benchmark/tasks/<bundle>/ where that directory has no __init__.py. No package code imports it; Harness and Apps run it as its own process. - In a bundle program, importlib.util.module_from_spec calls are admitted. Name-based lookups still fail there (__import__, import_module, module_from_spec imported by name, a find_spec call), and so do static imports of other cores. The same call fails everywhere else. - EVALUATOR_LOADERS and its per-function shape check are removed. first_wave_report().runtime_execution is computed: every bundle program that loads by path. - Gate tests cover both sides: any bundle and file name is admitted, name-based lookups and a static Knowledge import fail, a finder class that only defines find_spec stays admitted, and the same call fails in a deeper bundle file, a bundle with __init__.py, other Benchmark modules, Harness, Apps and knowledge/data scripts. - docs/module-layout.md and CHANGELOG describe the rule. No task bundle code changes. Refs #412 Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author) Signed-off-by: GUOGUO <xwdong1998@163.com>
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent incremental review of bcbfbde6a593ba7d3f1f06d9b30b7f2ebfb096a8 → 2d0fa299473c31a88bae94f9c3b9b8c59b99c559, base c6678fa8df0e2ab423587eba06151e8386b5b9ce. Non-voting COMMENT: a new static-policy gap remains; not approval.
This is a substantive policy expansion, not just cleanup: the two exact task/function/spec declarations are replaced by a location rule permitting qualified module_from_spec calls in any direct bundle file, including multiple calls and module-scope calls. The updated tests intentionally accept those cases. I assessed that proposed rule against actual source and an isolated normative documentation view, without reading other review opinions or author result claims.
1. The new find_spec prohibition misses ordinary import aliases.
tests/_gate_policy_support.py:380–397 rejects a call only when its unparsed function is exactly find_spec or importlib.util.find_spec. The subsequent name/attribute/import checks do not include find_spec. Thus this proposed fixture in benchmark/tasks/new_task/evaluate.py bypasses the documented name-lookup prohibition by static inspection:
import importlib.util
from importlib.util import find_spec as resolve
spec = resolve("inferrsi.knowledge")
module = importlib.util.module_from_spec(spec)The ImportFrom's original name is not in DYNAMIC_IMPORT_NAMES, resolve is not in NAME_LOOKUPS, and the qualified module_from_spec call is admitted by location. import importlib.util as util; util.find_spec(...) has the same gap. These are normal import aliases, not the explicitly excluded getattr/eval/runpy mechanisms. Add negative fixtures for both forms and either resolve their bindings or conservatively reject these imported lookup uses. Keep the positive test for defining a finder method: a method definition alone is not a lookup call.
This conclusion is a direct AST-branch trace; I did not execute either fixture or claim a measured runtime exploit.
2. Do not equate “no init.py” with “cannot be imported / runs only as a separate process.”
The new predicate checks directory shape only; same-core imports remain allowed, and admitted loads remain admitted when reached through another module's closure. For example, a new identifier-named benchmark/tasks/new_task/evaluate.py can be statically imported by Benchmark package code without the predicate checking that incoming edge. Python supports namespace subpackages without init.py. The source comment's process-only/non-importable premise is therefore not established by this gate. If that separation is required, test and reject package-code imports of admitted bundle programs; otherwise describe this as a location-based static policy rather than an import/process isolation guarantee.
Evidence and remaining scope: I inspected the changed gate, both affected test files and the safe documentation contract. The unchanged evaluator behavior tests do not establish the widened allowance for every future bundle. The original detailed acceptance/provenance gap remains; the broader proposal cannot itself prove authorization to replace the previously scoped two-loader rule.
Ready run 44232306814 currently has both full-check jobs and Windows compatibility running, macOS queued, and installed-package 3.10/3.11, dashboard-browser and pr-references successful. These are job metadata, not a local test run or evidence for the missing alias fixtures. No repository code/tests/lint were executed locally because no independently trusted environment was available. No source, merge, policy setting or stopped experiment was changed.
|
New head: This revises Kind C after an owner decision on #412 (#412 (comment)), which replaces point 3 of the earlier record:
Gate tests cover both sides. Admitted: a by-path load in any bundle and file name, including a new bundle name, with one call, two calls or at module level, and a finder class that only defines Local (Windows 11, CPython 3.13.5): the gate, compatibility-shell, forwarder, campaign-record, first-wave policy, root-inventory and evaluator-loader tests gave 171 passed and 329 subtests. The 18 rewritten gate tests passed after I fixed one assertion: on a synthetic tree without the other cores, the report keeps their The gate cannot see which path a bundle program loads, which was equally true of the two named loaders. The rule states where by-path loading is allowed; it is not a sandbox, and no reviewer has accepted it yet. |
…ograms (R3) Review (zengchuang-hw, 2d0fa29), two static-policy gaps in the bundle program rule: 1. find_spec was rejected only as a call spelled `find_spec(...)` or `importlib.util.find_spec(...)`. An alias (`from importlib.util import find_spec as resolve`, `import importlib.util as util; util.find_spec(...)`) passed. In a bundle program find_spec is now handled like the other dynamic-import names: any name, attribute or import of it fails, so aliases fail too. A finder class that only defines a find_spec method stays admitted. 2. "No __init__.py" did not make a bundle program unimportable: Python can import it through a namespace package. The gate now rejects any import of a bundle program from package code, Benchmark, Apps or another bundle, so an admitted by-path load cannot join another module's closure. The real tree has no such import. The source comment, docs/module-layout.md and CHANGELOG no longer claim the directory cannot be imported, and call the rule a location-based static policy, not a process or sandbox guarantee. New negative fixtures cover both alias forms, PathFinder.find_spec, module_from_spec reached through an alias, and absolute, relative, Apps and cross-bundle imports of an identifier-named bundle program. Refs #412 Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author) Signed-off-by: GUOGUO <xwdong1998@163.com>
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent incremental review of 2d0fa299473c31a88bae94f9c3b9b8c59b99c559 → 2e2de0462296bc1188a37c058f33c9ac076f7a1d, base c6678fa8df0e2ab423587eba06151e8386b5b9ce. Non-voting COMMENT, not approval.
The two concrete static-policy findings in my previous review are addressed in the changed source:
- Ordinary find_spec aliases: the updated scanner now includes find_spec in the bundle-specific prohibited names and checks Name, Attribute and the original ImportFrom name. This catches both previously reported alias forms without needing to resolve the alias's later call. The new fixtures also cover PathFinder.find_spec. The positive fixture defining a find_spec method remains, consistent with the AST distinction between a function definition's string name and a lookup expression.
- Imports of namespace-bundle programs: the report now checks incoming graph edges, independently of the ordinary same-core/Apps allowance. The five new fixtures cover absolute from-import, direct import, relative import, Apps and another bundle program. Tracing _direct_targets/_edges shows those fixtures resolve to the prohibited target. This addresses the earlier ordinary static-import counterexample; it does not establish runtime sandboxing or defeat mechanisms the scanner explicitly leaves out of scope.
I inspected the actual changed helper and regression-test source, plus an isolated normative view of the documentation delta. The new documentation's no-import and alias restrictions agree with these specific source changes. These are static conclusions, not claims that I executed the new fixtures. The source now explicitly recognizes namespace-package importability and labels the rule location-based rather than a process/sandbox guarantee.
At this check, ready run 44236885989 has successful installed-package 3.10/3.11, pr-references, dashboard-browser and Windows compatibility jobs; both full-check jobs and macOS are running. These are job-status observations, not completed full-suite evidence for the new regressions. Previous-head green runs do not prove this head passes.
The original detailed acceptance/provenance gap and the distinction between the proposed widened location rule and the formerly scoped two-loader rule remain. I therefore do not approve or claim Issue #412 complete. No repository code/tests/lint were executed locally because an independently trusted execution environment was unavailable. No source, merge, policy setting or stopped experiment was changed.
|
New head: Both points from the review of
The owner decision on #412 (issuecomment-6968396086) is edited to match, with a note that says what the first version got wrong. Local evidence on the tree committed as
The ready lane on |
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent CI follow-up at unchanged head 2e2de0462296bc1188a37c058f33c9ac076f7a1d, base c6678fa8df0e2ab423587eba06151e8386b5b9ce. Non-voting COMMENT, not approval.
The pending execution evidence in my previous review is now available:
- Actual full-check logs for Python 3.10 and Python 3.11 each show successful Ruff and 5148 passed, 36 skipped, 1 warning, 671 subtests passed.
- Both checked out the PR merge ref at
173793b1365ca4733b07afffe4f17bf2a767f470, not a bare-head tree. I fetched that tested merge'stests/_gate_policy_support.pyandtests/test_target_module_import_gate.py: both match the reviewed head exactly. - The matching test file contains the new alias/PathFinder negative cases and all five incoming bundle-program import cases, as well as the positive finder-method definition case. Both jobs ran
pytest -rs; the tested pytest configuration selectstestswith quiet output, and neither skip summary lists this test module. This supplies full-suite integration evidence for the regressions rather than inferring coverage merely from a passing-test-count increase. The quiet logs do not expose an individual verbose PASS line for each parametrized case. - All eight jobs in ready run 44236885989 now report success.
The two prior concrete static findings remain resolved, now with this relevant integration evidence; I am not reopening them as current bugs. The original detailed acceptance/provenance gap and evidence needed to establish acceptance of the widened location-based policy remain. Green CI alone does not resolve those scope questions or establish runtime sandboxing, so this is not approval or an Issue #412 completion claim.
No local repository tests/lint, fresh bare-head/base execution or model/device experiments were run. The 36 full-check skips remain limitations of these runs; other jobs' success is reported as metadata, not as proof that every skipped case ran elsewhere. No source, merge, policy setting or stopped experiment was changed.
Dong1017
left a comment
There was a problem hiding this comment.
PR review verdict — PASS
This is a self-review by Dong-bot (authorized delegate). The PR author is Dong1017, so this is not an independent approval. The ruleset still needs two independent write-access Approves, and none exists on this head.
Reviewed base c6678fa8df0e2ab423587eba06151e8386b5b9ce and head 2e2de0462296bc1188a37c058f33c9ac076f7a1d. This head follows bcbfbde6 and 2d0fa299.
Review state on this head
- There is no CHANGES_REQUESTED outstanding in the latest per-login decision set.
- There is no formal APPROVED review on this exact head.
- The only formal reviews on this head are two non-voting COMMENTs from @zengchuang-hw. The cursor[bot] review is dismissed and was for
bcbfbde6, so it is not counted here.
Earlier findings
- Assessor and arguments exception reason across the child transport. The
36384866blocker is fixed. The fixes are096f93c, which addedreason, andbcbfbde, which replaced it withtype_name. @zengchuang-hw'sbcbfbde6review confirmed this at source level. The regression runs through the real start/round path and checks the observation reason andcoordinator-error.json. find_specaliases in bundle programs. The2d0fa299review found that aliases such asfrom importlib.util import find_spec as resolvewere missed. @zengchuang-hw's review of2e2de046says this is fixed.find_specis now rejected as a Name, an Attribute and an ImportFrom name, and thePathFinder.find_specfixture is covered. A finder class that only definesfind_specstays admitted.- Imports of a bundle program through a namespace package. The same review says this is fixed. The gate now reports incoming edges into bundle programs. Five fixtures cover absolute, direct, relative, Apps and cross-bundle imports.
- Campaign-record closure test. The
5ee6912failure was fixed in70f6b56, and CI has been green since.
CI evidence
- @zengchuang-hw's CI follow-up on this head reports
full-check3.10 and 3.11 at 5148 passed, 36 skipped and 671 subtests, with Ruff successful. - All 8 jobs in ready run 44236885989 succeeded, per that review.
- Those runs used the PR merge ref, not a bare-head checkout. @zengchuang-hw confirmed that
tests/_gate_policy_support.pyandtests/test_target_module_import_gate.pymatch this head exactly.
Other points
- The
knowledge-model-policyblock in the PR body carrieshead_sha2e2de046…, which matches the current head. - The PR body says the Kind C rule is an owner policy decision on #412 (issuecomment-6968396086). It also says passing tests are not a reviewer acceptance of that rule. I take that as an accurate description of what this PR claims.
- The gate result is a static, location-based policy. It is not a sandbox or a runtime isolation guarantee, and the PR body says so.
Not verified here
- I did not read the full diff, because none was seeded for this mention. This verdict rests on the PR body, the thread, the formal reviews and the CI evidence in them.
- I ran no tests and did not re-run the gate.
- No native Windows Job run, NPU or H200 comparison, or old-bundle migration was verified.
Next step: an independent maintainer Approve is still required before merge.
Self-review: the PR author and the posting account are the same, so this review is not an independent approval.
Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).
zengchuang-hw
left a comment
There was a problem hiding this comment.
Follow-up to the direct review request in review 5951002790 (2026-11-18 04:53:46 UTC). Reviewed head remains 2e2de0462296bc1188a37c058f33c9ac076f7a1d, base c6678fa8df0e2ab423587eba06151e8386b5b9ce. Non-voting COMMENT, not approval.
I performed a fresh bounded check rather than treating this new request as already answered merely because the head is unchanged. The actual gate helper and regression-test file still exactly match the versions independently reviewed previously, and all eight jobs in ready run 44236885989 remain successful. The verified full-check evidence and tested-merge/source correspondence are recorded in my CI follow-up.
My technical conclusion is unchanged: the ordinary find_spec-alias and incoming bundle-program import counterexamples are addressed, with matching regression coverage in the successful integration runs. Neither is a current defect in this reviewed revision.
To preserve independent review, I used the isolated request metadata, the earlier isolated requirements, actual source and machine evidence, not another review's opinions or attributed conclusions. No additional substantive requirement could be safely separated from this new source, so I cannot claim to have adjudicated any further semantic clarification it may contain. A standalone normative statement of the intended acceptance criteria and whether the wider location-based rule replaces the earlier two-loader scope would make that remaining assessment possible.
The original detailed acceptance/provenance and widened-scope evidence gaps therefore remain; this follow-up does not establish acceptance or completion of Issue #412. No local repository execution, source changes, merge, policy changes or resumed experiments were performed.
|
The scope question in the reviews of The head is unchanged at |
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent requirement-change follow-up at unchanged head 2e2de0462296bc1188a37c058f33c9ac076f7a1d, base c6678fa8df0e2ab423587eba06151e8386b5b9ce. Non-voting COMMENT; no approval or Issue-completion claim.
Compared with my prior review, the normative Acceptance section of #412 now explicitly states that the location-based rule replaces the earlier exact two-loader/path/function/count scope. I verified the current Issue body and its author (Dong1017), rather than relying on the announcement or another review's interpretation.
What this resolves: the request for a standalone, unambiguous description of the intended wider static policy is now answered in the Issue body. I no longer treat the replacement wording itself as missing. This is a requirement-document change, not a new code revision or fresh test execution.
I independently re-read the current helper and regression source:
_is_bundle_programselects direct files under a task-bundle directory with no corresponding initializer module; it does not select an exact task/function/count list._by_path_loadsadmits the literalimportlib.util.module_from_speccall there; other detected dynamic lookups, including ordinaryfind_specaliases and PathFinder forms, remain rejected.- Incoming static edges to those bundle programs are rejected separately from same-core/Apps allowances. The five incoming-import fixtures and the non-package/deeper-file/other-location negative fixtures remain present.
- The real-tree test retains empty blockers, zero Harness-to-Infra/Model paths and the separate two-entry
runtime_executionreport. Those two entries describe the current tree, not a hard limit on future bundle files.
Thus the code's location-based static interpretation matches the newly explicit normative text. The two previously reported concrete scanner findings remain resolved; I am not reopening them. The current-head checks remain successful. My CI evidence review records the tested merge, matching gate/test bytes and the two Linux full-check results; this round refreshed status and source, not those executions.
What remains separate: the body calls this an owner decision, but the retrieved Issue is authored by the PR author and the attributed decision is not itself independent authentication of the maintainer's acceptance of a widened policy. I can assess the explicit proposed scope without converting that attribution into new authority or clearing the earlier acceptance-provenance gap. Before approval/completion, retain a verifiable acceptance record from the authorized requirements owner for this replacement. This is not a request to repeat permission for routine reviews.
Regardless of acceptance, this scanner does not sandbox candidate code or prove runtime import/network/filesystem isolation. The body now correctly says so. No local repository code/tests/lint, bare-head/base replay, device experiment or policy-setting change was performed in this round; no other reviewers' opinions were used. There is no additional code-change demand or new optional suggestion from this requirement-only recheck.
zengchuang-hw
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS for the independently reviewed R3 static-policy change.
Reviewed base c6678fa8df0e2ab423587eba06151e8386b5b9ce, exact head 2e2de0462296bc1188a37c058f33c9ac076f7a1d. Incremental range from my previous review is the same head: this round resolves the remaining requirement-source question, not a code change.
Issue-owner account confirmation now explicitly adopts the standalone location-based acceptance rule. My independent Issue follow-up records the exact scope and account-attribution limits. This closes the previous request for a direct requirements-owner acceptance record; it is not new authorization to merge or change security settings.
I re-read the unchanged gate helper and regression tests. The intended direct non-package bundle-file rule is explicit; literal by-path construction is distinguished from other detected dynamic lookups; incoming package-to-bundle-program imports are rejected. The ordinary find_spec-alias and incoming-import counterexamples from my earlier review remain addressed. No unresolved concrete code finding remains within this reviewed static-policy scope.
Validation: reopened the Python 3.10/3.11 full-check logs, each Ruff successful and 5148 passed, 36 skipped, 1 warning, 671 subtests passed. Tested merge is 173793b1365ca4733b07afffe4f17bf2a767f470; the gate/helper test content matches this head, as verified in my earlier CI review and unchanged since. Current required check records remain successful. Windows/macOS logs show their eight required host tests passing; Windows also passes the 25-case declared group. These are CI executions, not local reruns. Installed-package/browser success is check metadata, not a claim that all 36 optional skips were run elsewhere.
Approval applies to this PR's static import-policy implementation, not arbitrary runtime sandboxing, whole-Issue completion, real-model performance, external-source acceptance or the skipped optional environments. No local repository code/tests, experiment, source change or merge was performed. No other reviewers' opinions were used. No additional blocking or non-blocking change request is introduced in this requirement-only follow-up.
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base c6678fa8df0e2ab423587eba06151e8386b5b9ce and head 2e2de0462296bc1188a37c058f33c9ac076f7a1d.
Prior assessor-reason blocker (36384866) remains cleared via TaskProtocolError.type_name on the real child path.
Tip delta since dismissed Approve bcbfbde6: location-based by-path policy (2d0fa299) plus find_spec-alias and incoming bundle-program import rejection (2e2de04); both zengchuang findings at 2d0fa299 cleared; #412 owner/steward acceptance of the location rule confirmed.
first_wave_report() empty violations/blockers/harness→model-infra; isolation flags true; Ready run 44236885989 SUCCESS (full-check 3.10/3.11).
No outstanding blockers.
|
Posted by an authorized steward agent on behalf of @Dong1017 (hosted Dong-bot on Independent tip re-review at
Merged (squash) → |
zengchuang-hw
left a comment
There was a problem hiding this comment.
Incremental follow-up to the new mention in 6973422418.
Verified independently: the PR is merged (2026-11-18 08:38:28 UTC), with reviewed head 2e2de0462296bc1188a37c058f33c9ac076f7a1d landed as 91c150589567292292c1a37d7ccbebd7dd1015b6. The current main contains that commit. The landed gate helper and regression-test file match the reviewed bytes exactly. My existing approval remains attached to the reviewed head; no code delta or new defect in those checked files was identified.
The landed commit currently shows successful declaration/policy-manifest checks. These are policy checks, not a new product-test run. This follow-up adds landing/content-correspondence evidence only; the previous test evidence and limitations remain as recorded in my approval. The source's review links were not used as technical evidence or read for others' opinions.
No new approval is necessary. No local tests or repository code were executed, and I did not perform the merge. This confirms landing of the reviewed static-policy change, not runtime sandboxing or acceptance of the whole parent Issue.
Scope
Refs #412
R3, the last item of #412. It follows the merged C1, C2, R1, A1, A2, M1 and R2 PRs and starts from
mainc6678fa8df0e2ab423587eba06151e8386b5b9ce(R2, #478). It implements the design in #412 (comment) and answers the four conditions in #412 (comment) (see "Design conditions" below).On this tree
first_wave_report()returnsviolations == (),harness_model_infra == (),blockers == (),new_core_closure_isolated is Trueandisolation_complete is True.Kind A: the Harness task protocol runs in a child interpreter
harness/task_protocol.pyno longer creates a module or executesoptimization.pyin the Harness process.task_protocol.run(root, protocol_sha256, operation, **arguments)runs the newharness/protocol_child.pyaspython -I -Bfor eachvalidate_arguments,argumentsandassesscall:local_process._supervise, in its own POSIX process group or Windows Job, with a 120 s deadline (PROTOCOL_TIMEOUT_SECONDS). On timeout, and after a normal exit, every remaining member of the group or Job is stopped. A run whose domain is not confirmed empty fails with kindcleanup.inferrsi. It reads the protocol bytes once, checks them againstprotocol_sha256, and compiles exactly those bytes. No bytecode cache is read or written.request,identity,protocol_exception(with the original exception type),result(NaN or a non-JSON value),malformed_response(bad JSON, duplicate key, NaN or infinity),invalid_response(schema, version or operation mismatch, wrong result type),exit_status,timeout,launch,cleanupandoutput_limit. A protocolSystemExitis raised again with its exit code after the child's stderr is copied, so an argparse usage error insession startstill exits 2 with its message.preparevalidates through the child, freezes the resources and checks that the frozenoptimization.pydigest equals the validated one. Every laterargumentsandassesscall passesplan["frozen_files"]["optimization.py"], and the parent and the child each refuse other bytes.harness/protocol_child.pyjoins the optimization session's runtime identity, andharness/task_protocol.pychanged, so a session started before this change refuses to resume after it.Kind B: no bundle loads a sibling module by path
moe-forward-npu:processes.pyis folded intobenchmark.pywithout code changes (MARKER,_members,cleanup) and leavesTASK_FILES.qwen3-tts-npu:evaluation.pybecomesbenchmark.py(git rename) and the 98-line CLI joins it. The prepared bundle no longer containsevaluation.py.policy.jsonpins the grader digest and every bundle file, so a bundle prepared before R3 failsload_preparedand must be prepared again. Retained results are not rewritten.Kind C: by-path loading in task bundle programs, by file location
The MoE grader and the Qwen21 evaluator must call the submitted
computein their own process to time it, so they load the frozen candidate file by path. Turning that into a plainimportwith asys.pathchange would only hide the load from the gate, which #412 rules out. The gate therefore decides by file location and names no task (tests/_gate_policy_support.py):benchmark/tasks/<bundle>/where that directory has no__init__.py. Python could still import it through a namespace package, so the gate rejects every import of a bundle program from package code; Harness and Apps run it as its own process.importlib.util.module_from_speccalls are admitted. Name-based lookups still fail there:__import__,importlib.import_module,module_from_specreached by another spelling, andfind_specin any spelling, including an alias. Static imports of Knowledge, Harness or Apps fail as in any Benchmark module.__init__.py,knowledge/data/**scripts, other Benchmark modules, Harness and Apps.first_wave_report().runtime_executionlists each bundle program that loads by path (today the MoE grader and the Qwen21 evaluator).isolation_completeis a static import-closure result and is documented as such.RESOURCE_LOADERStable and theruntime resource loadersblocker are deleted.Design conditions from zengchuang-hw (issuecomment-6948387498)
schema_version1 and exact per-operation argument sets inprotocol_child.check_request; result types checked per operation; 16 MiB / 4 KiB bounds; 120 s deadline through_supervise; the eleven kinds above; descriptor 1 redirected before the protocol runs.test_task_protocol_child.pycovers each kind with a real child, including deadline and post-exit descendant cleanup.test_optimization_session.pydrivesvalidate_arguments,argumentsandassessthrough the real CLIstartandroundpath.-I -Bis not a sandbox; pin exact bytes before execution; refuse old resume identities; mutation testsdocs/module-layout.mdand CHANGELOG. The parent and the child each check the SHA-256 before compiling;test_changed_protocol_bytes_never_runandtest_child_checks_the_digest_of_the_bytes_it_compilesshow changed bytes never run. The runtime identity refuses sessions from before this change.2d0fa29after an owner decision (#412, issuecomment-6968396086), and tightened at2e2de04after zengchuang-hw's review of2d0fa29: the rule is by file location, not by task, function or count, so it no longer names any task. Negative controls: name-based lookups next to a by-path load (import_module,__import__,module_from_specimported by name or reached through an alias,find_specin any spelling: direct,importlib.util.find_spec,from importlib.util import find_spec as resolve,import importlib.util as util; util.find_spec(...),PathFinder.find_spec); absolute, relative, Apps and cross-bundle imports of a bundle program; a static Knowledge import from a bundle program; a bundle directory given an__init__.py; a deeper file in a bundle; the same call inbenchmark/registry.py,benchmark/tasks.py,harness/task_protocol.py, another Harness module, Apps and aknowledge/datascript; the retired tables returning. A finder class that only definesfind_spec(as in the Qwen21 and TTS import guards) stays admitted.IsolationReport.runtime_executionis computed and separate from both static flags.__init__.py, adjacent imports and freshnesstest_evaluator_loaders.py: each loader returns a fresh module on every call and never registers it insys.modules; the MoEloadrequires a callablecompute; Qwen21candidate_functionrejects a missing, non-callable or wrong-signaturecomputeand a syntax error. File identity stays with the callers that already verify it (_adjudicatere-hashesidentity.jsonfor MoE; Qwen21mainre-hashes its snapshot and the frozenassessre-checks it against the frozen task). The package and adjacent-module controls are in the gate tests above.Tests
tests/test_task_protocol_child.py(33 tests) andtests/test_evaluator_loaders.py.tests/test_target_module_import_gate.py; the real-tree expectations in that file,test_knowledge_compatibility_shells.pyandtest_legacy_run_forwarder_gate.pynow requireblockers == ()and isolation.test_moe_benchmark.py,test_tts_benchmark.py,test_tts_standalone.py,test_engineering_benchmark_plugins.py(the prepared TTS bundle has noevaluation.py) and the runtime-identity test.Evidence
src tests toolsandgit diff --check: pass at5ee6912,70f6b56and3638486.full-checkat5ee6912(run 44158685697): 3.10 and 3.11 each 1 failed, 4993 passed, 29 skipped, 671 subtests. The one failure wastest_campaign_record_injection.py::test_harness_records_closure_holds_as_declared_entries, which still assertednot report.isolation_completeon the real tree; fixed in70f6b56.70f6b56(run 44170321274): all 11 checks succeeded, includingfull-check3.10/3.11.3638486:test_task_protocol_child.py37 passed. The earlier local run oftest_task_protocol_child.py,test_campaign_record_injection.pyandtest_target_module_import_gate.pyat70f6b56gave 79 passed, 43 subtests.3638486changes onlytests/test_task_protocol_child.py.3638486(run 44179562000): all 11 checks succeeded;full-check3.10/3.11 each 4998 passed, 29 skipped, 671 subtests.096f93c(cursor review 5948608044): a protocol exception now settles with its own"<Type>: <message>". Locallytest_optimization_session.py(with the two new real start/round regressions) andtest_task_protocol_child.pypass. On this laptoptest_project_optimization.py::test_final_reserve_blocks_search_but_allows_finalfails withOptimization deadline exhaustedat096f93cand also at the unchanged3638486, wherefull-checkpassed, so it is a laptop-timing failure.test_project_lifecycle_never_imports_benchmarkpassed when rerun alone. The ready lane on096f93cis the full-suite proof.096f93c(run 44215223299): all checks succeeded;full-check3.10/3.11 each 5074 passed, 31 skipped, 671 subtests, on GitHub's merge tree withmainc37832c.bcbfbde(zengchuang-hw review at096f93c): anargumentsfailure keeps the protocol's own type incoordinator-error.json, as before R3. Locallytest_optimization_session.pyandtest_task_protocol_child.pygive 61 passed. The ready lane onbcbfbdeis the full-suite proof.bcbfbde(run 44227239166): all checks succeeded;full-check3.10/3.11 each 5075 passed, 31 skipped, 671 subtests, on GitHub's merge tree.2d0fa29(owner decision issuecomment-6968396086): locally the gate, compatibility-shell, forwarder, campaign-record, first-wave policy, root-inventory and evaluator-loader tests give 171 passed with 329 subtests plus the 18 rewritten gate tests passed after one test-assertion fix; Ruff 0.17.5 passes. No task bundle code changed.2e2de04(zengchuang-hw review at2d0fa29): in a bundle programfind_specfails in any spelling, including aliases, and any import of a bundle program from package code fails. Locallytest_target_module_import_gate.pygives 35 passed with 43 subtests.test_first_wave_isolation_policy.py,test_knowledge_compatibility_shells.py,test_legacy_run_forwarder_gate.py,test_campaign_record_injection.py,test_package_root_inventory.pyandtest_evaluator_loaders.pygive 147 passed with 286 subtests. Ruff 0.17.5 andgit diff --checkpass. Only the gate, its tests,docs/module-layout.mdand CHANGELOG changed. The ready lane on2e2de04is the full-suite proof.2e2de04(run 44236885989): all checks succeeded;full-check3.10/3.11 each 5148 passed, 36 skipped, 671 subtests, on GitHub's merge tree.5ee6912:test_target_module_import_gate.py,test_knowledge_compatibility_shells.py,test_legacy_run_forwarder_gate.pyandtest_package_root_inventory.pyafter the last doc edit: 88 passed, 43 subtests.test_task_protocol_child.py,test_evaluator_loaders.py,test_tts_standalone.pyand the gate file: all passed, including the deadline and descendant-cleanup tests inside a Windows Job. The same run had one doc-wording failure, fixed before the commit and covered by the 88-test run above.test_optimization_session.py,test_qwen_optimization_protocol.py,test_tts_benchmark.py,test_moe_benchmark.py,test_first_wave_isolation_policy.py,test_package_root_inventory.py,test_knowledge_compatibility_shells.pyandtest_legacy_run_forwarder_gate.pybefore the doc edits: 219 passed, 2 skipped, 286 subtests.test_engineering_benchmark_plugins.pycannot run in this sparse local checkout (noplugins/directory);full-checkruns it.full-check3.10/3.11,installed-packageandhost-compatibilityjobs. No separate H200 candidate-versus-maincomparison was run.Not claimed
--check-onlypath and the MoE no-device contract tests exercise the loaders.runpyandexecremain outside the gate's scan scope, as before; no loader moved to them.Provenance
Commits on
mainc6678fa: the change (5ee6912); a fix (70f6b56) for the onefull-checkfailure (test_campaign_record_injection.pystill pinned the old not-isolated real tree, a test this PR missed) and for the review point on diagnostics; and a test-only follow-up (3638486) that asserts stderr forwarding on the timeout, nonzero-exit and malformed-envelope paths and scratch cleanup; and a fix (096f93c, cursor review 5948608044) that keeps a protocol exception's own"<Type>: <message>"as the settlement reason and the CLI message; and a fix (bcbfbde, zengchuang-hw review at096f93c) that keeps the protocol's own exception type incoordinator-error.json; and a gate change (2d0fa29, owner decision on #412) that admits by-path loading by bundle location instead of two named tasks; and a fix (2e2de04, zengchuang-hw review at2d0fa29) that rejectsfind_specaliases and any package import of a bundle program. No rebase, no amend, no force push, no skipped hooks. The code, tests and docs were written in a Claude Code session reported asclaude-opus-5-5, effortxhigh, confirmed by the owner. This is client/session attribution, not a backend attestation. The changed paths are outside theknowledge-model-policyprotected scope, so the declaration below is disclosure only.