Skip to content

refactor(harness,benchmark): close the five runtime resource loaders (R3) - #486

Merged
Dong1017 merged 7 commits into
mainfrom
impl/r3-execution-boundaries
Nov 18, 2026
Merged

Dong1017 merged 7 commits into
mainfrom
impl/r3-execution-boundaries

Conversation

@Dong1017

@Dong1017 Dong1017 commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Scope

Refs #412

R3, the last item of #412. It follows the merged C1, C2, R1, A1, A2, M1 and R2 PRs and starts from main c6678fa8df0e2ab423587eba06151e8386b5b9ce (R2, #478). It implements the design in #412 (comment) and answers the four conditions in #412 (comment) (see "Design conditions" below).

On this tree first_wave_report() returns violations == (), harness_model_infra == (), blockers == (), new_core_closure_isolated is True and isolation_complete is True.

Kind A: the Harness task protocol runs in a child interpreter

harness/task_protocol.py no longer creates a module or executes optimization.py in the Harness process. task_protocol.run(root, protocol_sha256, operation, **arguments) runs the new harness/protocol_child.py as python -I -B for each validate_arguments, arguments and assess call:

  • The child starts through local_process._supervise, in its own POSIX process group or Windows Job, with a 120 s deadline (PROTOCOL_TIMEOUT_SECONDS). On timeout, and after a normal exit, every remaining member of the group or Job is stopped. A run whose domain is not confirmed empty fails with kind cleanup.
  • The request is one version-1 JSON object written to a private scratch file, which is also the child's working directory. Paths are absolute strings.
  • The child imports only the standard library and never loads inferrsi. It reads the protocol bytes once, checks them against protocol_sha256, and compiles exactly those bytes. No bytecode cache is read or written.
  • Descriptor 1 carries only the response envelope. The child keeps a private copy of it and points descriptor 1 at stderr before the protocol runs, so Python prints, raw descriptor writes and grandchildren cannot reach the envelope. After every operation, whatever its result, Harness copies the last 4 KiB of the child's stderr to its own stderr and deletes the scratch directory. Harness reads at most 16 MiB of envelope. Both limits apply when Harness reads the child's output files, not while the child writes them; the deadline bounds the run.
  • Each failure keeps its own kind: request, identity, protocol_exception (with the original exception type), result (NaN or a non-JSON value), malformed_response (bad JSON, duplicate key, NaN or infinity), invalid_response (schema, version or operation mismatch, wrong result type), exit_status, timeout, launch, cleanup and output_limit. A protocol SystemExit is raised again with its exit code after the child's stderr is copied, so an argparse usage error in session start still exits 2 with its message.
  • prepare validates through the child, freezes the resources and checks that the frozen optimization.py digest equals the validated one. Every later arguments and assess call passes plan["frozen_files"]["optimization.py"], and the parent and the child each refuse other bytes.
  • harness/protocol_child.py joins the optimization session's runtime identity, and harness/task_protocol.py changed, so a session started before this change refuses to resume after it.

Kind B: no bundle loads a sibling module by path

  • moe-forward-npu: processes.py is folded into benchmark.py without code changes (MARKER, _members, cleanup) and leaves TASK_FILES.
  • qwen3-tts-npu: evaluation.py becomes benchmark.py (git rename) and the 98-line CLI joins it. The prepared bundle no longer contains evaluation.py. policy.json pins the grader digest and every bundle file, so a bundle prepared before R3 fails load_prepared and must be prepared again. Retained results are not rewritten.

Kind C: by-path loading in task bundle programs, by file location

The MoE grader and the Qwen21 evaluator must call the submitted compute in their own process to time it, so they load the frozen candidate file by path. Turning that into a plain import with a sys.path change would only hide the load from the gate, which #412 rules out. The gate therefore decides by file location and names no task (tests/_gate_policy_support.py):

  • A task bundle program is a file directly inside benchmark/tasks/<bundle>/ where that directory has no __init__.py. Python could still import it through a namespace package, so the gate rejects every import of a bundle program from package code; Harness and Apps run it as its own process.
  • In a bundle program, importlib.util.module_from_spec calls are admitted. Name-based lookups still fail there: __import__, importlib.import_module, module_from_spec reached by another spelling, and find_spec in any spelling, including an alias. Static imports of Knowledge, Harness or Apps fail as in any Benchmark module.
  • Everywhere else the same call fails: a deeper file in a bundle, a bundle directory given an __init__.py, knowledge/data/** scripts, other Benchmark modules, Harness and Apps.
  • first_wave_report().runtime_execution lists each bundle program that loads by path (today the MoE grader and the Qwen21 evaluator). isolation_complete is a static import-closure result and is documented as such.
  • No table names a task or function, so a new bundle needs no gate change. The temporary RESOURCE_LOADERS table and the runtime resource loaders blocker are deleted.

Design conditions from zengchuang-hw (issuecomment-6948387498)

Condition How this PR meets it
1. Versioned per-operation contract; bounded stdout/stderr; deadline and termination; distinct failure kinds; stdout only the envelope; all three operations at their real callers; descendant cleanup schema_version 1 and exact per-operation argument sets in protocol_child.check_request; result types checked per operation; 16 MiB / 4 KiB bounds; 120 s deadline through _supervise; the eleven kinds above; descriptor 1 redirected before the protocol runs. test_task_protocol_child.py covers each kind with a real child, including deadline and post-exit descendant cleanup. test_optimization_session.py drives validate_arguments, arguments and assess through the real CLI start and round path.
2. -I -B is not a sandbox; pin exact bytes before execution; refuse old resume identities; mutation tests Stated in the module docstring, docs/module-layout.md and CHANGELOG. The parent and the child each check the SHA-256 before compiling; test_changed_protocol_bytes_never_run and test_child_checks_the_digest_of_the_bytes_it_compiles show changed bytes never run. The runtime identity refuses sessions from before this change.
3. The permanent rule is a policy change; exact path/function/count; adversarial controls; static vs runtime in the report Revised at 2d0fa29 after an owner decision (#412, issuecomment-6968396086), and tightened at 2e2de04 after zengchuang-hw's review of 2d0fa29: the rule is by file location, not by task, function or count, so it no longer names any task. Negative controls: name-based lookups next to a by-path load (import_module, __import__, module_from_spec imported by name or reached through an alias, find_spec in any spelling: direct, importlib.util.find_spec, from importlib.util import find_spec as resolve, import importlib.util as util; util.find_spec(...), PathFinder.find_spec); absolute, relative, Apps and cross-bundle imports of a bundle program; a static Knowledge import from a bundle program; a bundle directory given an __init__.py; a deeper file in a bundle; the same call in benchmark/registry.py, benchmark/tasks.py, harness/task_protocol.py, another Harness module, Apps and a knowledge/data script; the retired tables returning. A finder class that only defines find_spec (as in the Qwen21 and TTS import guards) stays admitted. IsolationReport.runtime_execution is computed and separate from both static flags.
4. Per-caller callable/identity checks; separate controls for registration, wrong path, __init__.py, adjacent imports and freshness test_evaluator_loaders.py: each loader returns a fresh module on every call and never registers it in sys.modules; the MoE load requires a callable compute; Qwen21 candidate_function rejects a missing, non-callable or wrong-signature compute and a syntax error. File identity stays with the callers that already verify it (_adjudicate re-hashes identity.json for MoE; Qwen21 main re-hashes its snapshot and the frozen assess re-checks it against the frozen task). The package and adjacent-module controls are in the gate tests above.

Tests

  • New tests/test_task_protocol_child.py (33 tests) and tests/test_evaluator_loaders.py.
  • Rewritten gate controls in tests/test_target_module_import_gate.py; the real-tree expectations in that file, test_knowledge_compatibility_shells.py and test_legacy_run_forwarder_gate.py now require blockers == () and isolation.
  • Updated consumers: test_moe_benchmark.py, test_tts_benchmark.py, test_tts_standalone.py, test_engineering_benchmark_plugins.py (the prepared TTS bundle has no evaluation.py) and the runtime-identity test.
  • No assertion, deadline or gate was weakened. The only expected values that changed are the gate results that R3 is meant to change.

Evidence

  • Ruff 0.17.5 on src tests tools and git diff --check: pass at 5ee6912, 70f6b56 and 3638486.
  • full-check at 5ee6912 (run 44158685697): 3.10 and 3.11 each 1 failed, 4993 passed, 29 skipped, 671 subtests. The one failure was test_campaign_record_injection.py::test_harness_records_closure_holds_as_declared_entries, which still asserted not report.isolation_complete on the real tree; fixed in 70f6b56.
  • Ready lane at 70f6b56 (run 44170321274): all 11 checks succeeded, including full-check 3.10/3.11.
  • Local at 3638486: test_task_protocol_child.py 37 passed. The earlier local run of test_task_protocol_child.py, test_campaign_record_injection.py and test_target_module_import_gate.py at 70f6b56 gave 79 passed, 43 subtests. 3638486 changes only tests/test_task_protocol_child.py.
  • Ready lane at 3638486 (run 44179562000): all 11 checks succeeded; full-check 3.10/3.11 each 4998 passed, 29 skipped, 671 subtests.
  • Fix 096f93c (cursor review 5948608044): a protocol exception now settles with its own "<Type>: <message>". Locally test_optimization_session.py (with the two new real start/round regressions) and test_task_protocol_child.py pass. On this laptop test_project_optimization.py::test_final_reserve_blocks_search_but_allows_final fails with Optimization deadline exhausted at 096f93c and also at the unchanged 3638486, where full-check passed, so it is a laptop-timing failure. test_project_lifecycle_never_imports_benchmark passed when rerun alone. The ready lane on 096f93c is the full-suite proof.
  • Ready lane at 096f93c (run 44215223299): all checks succeeded; full-check 3.10/3.11 each 5074 passed, 31 skipped, 671 subtests, on GitHub's merge tree with main c37832c.
  • Fix bcbfbde (zengchuang-hw review at 096f93c): an arguments failure keeps the protocol's own type in coordinator-error.json, as before R3. Locally test_optimization_session.py and test_task_protocol_child.py give 61 passed. The ready lane on bcbfbde is the full-suite proof.
  • Ready lane at bcbfbde (run 44227239166): all checks succeeded; full-check 3.10/3.11 each 5075 passed, 31 skipped, 671 subtests, on GitHub's merge tree.
  • Gate rule by location 2d0fa29 (owner decision issuecomment-6968396086): locally the gate, compatibility-shell, forwarder, campaign-record, first-wave policy, root-inventory and evaluator-loader tests give 171 passed with 329 subtests plus the 18 rewritten gate tests passed after one test-assertion fix; Ruff 0.17.5 passes. No task bundle code changed.
  • Fix 2e2de04 (zengchuang-hw review at 2d0fa29): in a bundle program find_spec fails in any spelling, including aliases, and any import of a bundle program from package code fails. Locally test_target_module_import_gate.py gives 35 passed with 43 subtests. test_first_wave_isolation_policy.py, test_knowledge_compatibility_shells.py, test_legacy_run_forwarder_gate.py, test_campaign_record_injection.py, test_package_root_inventory.py and test_evaluator_loaders.py give 147 passed with 286 subtests. Ruff 0.17.5 and git diff --check pass. Only the gate, its tests, docs/module-layout.md and CHANGELOG changed. The ready lane on 2e2de04 is the full-suite proof.
  • Ready lane at 2e2de04 (run 44236885989): all checks succeeded; full-check 3.10/3.11 each 5148 passed, 36 skipped, 671 subtests, on GitHub's merge tree.
  • Local (Windows 11, CPython 3.13.5), the tree committed as 5ee6912:
    • test_target_module_import_gate.py, test_knowledge_compatibility_shells.py, test_legacy_run_forwarder_gate.py and test_package_root_inventory.py after the last doc edit: 88 passed, 43 subtests.
    • test_task_protocol_child.py, test_evaluator_loaders.py, test_tts_standalone.py and the gate file: all passed, including the deadline and descendant-cleanup tests inside a Windows Job. The same run had one doc-wording failure, fixed before the commit and covered by the 88-test run above.
    • test_optimization_session.py, test_qwen_optimization_protocol.py, test_tts_benchmark.py, test_moe_benchmark.py, test_first_wave_isolation_policy.py, test_package_root_inventory.py, test_knowledge_compatibility_shells.py and test_legacy_run_forwarder_gate.py before the doc edits: 219 passed, 2 skipped, 286 subtests.
    • test_engineering_benchmark_plugins.py cannot run in this sparse local checkout (no plugins/ directory); full-check runs it.
  • Full suite: this PR's full-check 3.10/3.11, installed-package and host-compatibility jobs. No separate H200 candidate-versus-main comparison was run.

Not claimed

  • No candidate sandbox. The evaluators still run candidate code in their own process, and the protocol child is an interpreter and process boundary, not a filesystem or network sandbox.
  • The gate cannot see which path a bundle program loads. The rule states where by-path loading is allowed; the callers' identity checks bind the loaded files.
  • No NPU run. The NPU paths change only around the moved code; the Qwen21 CPU --check-only path and the MoE no-device contract tests exercise the loaders.
  • runpy and exec remain outside the gate's scan scope, as before; no loader moved to them.
  • No reviewer acceptance of the bundle-program rule; it is the owner's decision on task: R01 — retire legacy root paths and close core dependencies #412, and passing tests are not that review.

Provenance

Commits on main c6678fa: the change (5ee6912); a fix (70f6b56) for the one full-check failure (test_campaign_record_injection.py still pinned the old not-isolated real tree, a test this PR missed) and for the review point on diagnostics; and a test-only follow-up (3638486) that asserts stderr forwarding on the timeout, nonzero-exit and malformed-envelope paths and scratch cleanup; and a fix (096f93c, cursor review 5948608044) that keeps a protocol exception's own "<Type>: <message>" as the settlement reason and the CLI message; and a fix (bcbfbde, zengchuang-hw review at 096f93c) that keeps the protocol's own exception type in coordinator-error.json; and a gate change (2d0fa29, owner decision on #412) that admits by-path loading by bundle location instead of two named tasks; and a fix (2e2de04, zengchuang-hw review at 2d0fa29) that rejects find_spec aliases and any package import of a bundle program. No rebase, no amend, no force push, no skipped hooks. The code, tests and docs were written in a Claude Code session reported as claude-opus-5-5, effort xhigh, confirmed by the owner. This is client/session attribution, not a backend attestation. The changed paths are outside the knowledge-model-policy protected scope, so the declaration below is disclosure only.

{"schema_version":1,"head_sha":"2e2de0462296bc1188a37c058f33c9ac076f7a1d","human_only":false,"material_ai_authors":[{"tool":"Claude Code","model_id":"claude-opus-5-5","effort":"xhigh","role":"author"}]}

…(R3)

Close the last first-wave blocker. first_wave_report() now has no
violation and no blocker, and new_core_closure_isolated and
isolation_complete are both true.

- Harness no longer runs task protocol code in its own process. Each
  validate_arguments / arguments / assess call runs the new
  harness/protocol_child.py as `python -I -B` under a 120 s deadline,
  inside its own process group or Windows Job, and answers with one
  versioned strict JSON envelope on a private copy of stdout. The
  child imports only the standard library and compiles only the
  protocol bytes that match the frozen SHA-256. Failures stay distinct
  (request, identity, protocol_exception, result, malformed_response,
  invalid_response, exit_status, timeout, launch, cleanup, output_limit);
  a protocol SystemExit keeps its exit code, so argparse usage errors
  still exit 2. task_protocol.protocol() is replaced by run().
- moe-forward-npu folds processes.py into benchmark.py, and
  qwen3-tts-npu folds evaluation.py and its CLI into benchmark.py, so
  no bundle loads a sibling module by path.
- The MoE grader and Qwen21 evaluator keep their in-process candidate
  loaders under a permanent gate rule (EVALUATOR_LOADERS): one exact
  call each, only in a task bundle program, with negative controls for
  every other placement. first_wave_report().runtime_execution lists
  them, so static isolation is not read as runtime isolation.
- The optimization session's runtime identity adds
  harness/protocol_child.py.

Refs #412

Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author)
Signed-off-by: GUOGUO <xwdong1998@163.com>
@Dong1017 Dong1017 added area:harness Harness module: profile, A/B loop, quality gates, evidence reports kind:task Bounded delivery task with a delegated slice labels Nov 17, 2026

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent non-voting review: bounded checks passed; acceptance remains incomplete, not approved.

Reviewed head 5ee69128d228b921aa2446f3f0211057b22d56fa against base c6678fa8df0e2ab423587eba06151e8386b5b9ce. Repository/author/head were verified; I inspected actual executable changes and isolated original scope/documentation constraints without reading other reviews or author test/result summaries. No implementation contribution was made.

What the code and fresh tests establish

  • Harness calls a standalone protocol_child.py using -I -B, supervised process execution and versioned strict JSON. Parent and child check the protocol digest; the child compiles the bytes it hashed. Result shapes, duplicate/nonfinite JSON, usage exit2, typed failures, identity drift, process cleanup and runtime-identity changes have focused coverage.
  • AST comparison of relocated bodies found MoE _members/cleanup and all15 TTS evaluation functions unchanged after removing comments/docstrings. I inspected the remaining CLI/manifest/caller changes rather than treating the move alone as proof of compatibility.
  • The permanent evaluator rule records two exact loader sites. A direct gate invocation returned no violations/blockers/Harness→Model-or-Infra edges and both static isolation flags true, while separately listing the two in-process evaluator executions. That is a static-policy result under this new rule, not a sandbox or proof that candidate execution is safe.

Observed diagnostic behavior needing an explicit contract
task_protocol.run reads a stderr tail but writes it to the caller only for a SystemExit response. A controlled protocol printing a sentinel to stderr and returning normally produced no parent stderr; printing it then raising ValueError likewise produced no parent stderr, only the typed exception message. The temporary logs are deleted on return. The new test named for output going to stderr only asserts the returned result. The isolated proposed docs do not promise general diagnostic retention, so I am not inventing that as an accepted requirement or calling this a proven AC violation. Please explicitly decide/document/test whether diagnostics are intentionally discarded, forwarded, or retained; the transport currently does not preserve them for ordinary success/protocol exceptions. Similarly the16MiB response-size check is post-execution, not a live producer-output quota.

Checks actually executed

  1. Task-protocol child, evaluator loaders, target-module import gate and optimization session: 85 passed,43 subtests passed.
  2. MoE benchmark, TTS benchmark/standalone, engineering benchmark plugins, Knowledge compatibility shells and legacy-forwarder gate: 130 passed after correcting my source-import setup to an absolute PYTHONPATH.
    The initial second run had129 pass/1 fail: the fake launcher could not import inferrsi after changing cwd with my relative PYTHONPATH. The isolated rerun and then the entire130-test group passed with the corrected setup; I do not report this as a product defect. Each invocation had a pytest-plugin deprecation warning. Targeted Ruff had zero errors; git diff --check passed. Tests used synthetic/local CPU fixtures, not real model/device execution.

Exact-head CI currently reports success for lint-and-smoke, declarations, pr-references, installed-package3.10/3.11, Windows host and dashboard-browser. Both full-check versions are still running and macOS is queued. I did not run the full candidate-versus-base Linux suite/failure-ID comparison, local native Windows qualification, a clean installed-wheel matrix or device/model correctness/performance checks.

Acceptance gap: the three safely separable goals are supported by the inspection above, but the full original Issue412 protocol/gate acceptance contract cannot be independently recovered from the mixed later decisions. Proposed documentation does not authenticate authority to redefine acceptance. Resolve that provenance gap and pending full-check/platform evidence before treating this as complete approval or Issue412 closure.

AGENTS and review guidance were checked. Working tree is clean; no code, security settings, evidence, issue state, merge or deployment was changed, and no stopped experiment was resumed.

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent CI follow-up to my earlier bounded review, at unchanged head 5ee69128d228b921aa2446f3f0211057b22d56fa, base c6678fa8df0e2ab423587eba06151e8386b5b9ce. Non-voting COMMENT; this is not approval.

The two completed Linux full-check logs now provide a specific additional blocker:

  • Python 3.10 job 134306269863 and Python 3.11 job 134306269361 each report 1 failed, 4993 passed, 29 skipped, 671 subtests passed. Both fail tests/test_campaign_record_injection.py::test_harness_records_closure_holds_as_declared_entries, line 410: assert not report.isolation_complete.
  • I independently reran that exact test on isolated base and candidate trees with the same local Python 3.12 environment and absolute per-tree PYTHONPATH: base passes; candidate fails at the same assertion. The test file is unchanged between these pins. This is a reproduced candidate-versus-base test regression, not an attribution based solely on CI color.
  • A fresh direct computation with the same declared record entries gives base: zero violations, zero Harness-to-model/infra paths, blocker runtime resource loaders: 5, isolation_complete=False; candidate: zero violations, zero such paths, no blockers, isolation_complete=True, two separately reported runtime-execution entries. The failed assertion therefore conflicts with the changed static-gate reporting model. It does not demonstrate a newly introduced cross-core import or establish runtime sandboxing.

Please reconcile this retained test with the intended, explicitly scoped static-isolation contract while keeping assertions for declared-entry traversal and the separate runtime-execution limitations. Do not simply restore a false status or delete coverage to make CI green. The original detailed acceptance/provenance gap from my earlier review remains; this observation does not authorize changing acceptance criteria.

Checks in this follow-up are the two machine logs, exact failing-test base/head comparison, direct report-field comparison, and current PR head/base/author plus own-review deduplication. I did not rerun the whole suite locally, reproduce on local Python 3.10/3.11, or run model/device/service experiments. The earlier 215-test focused result remains an earlier result, not a new full-suite pass. No repository source changes or merge actions.

@Dong1017

Copy link
Copy Markdown
Collaborator Author

Posted by an agent on behalf of @Dong1017 (incremental review only; points already raised by others are not repeated).

PR review verdict — DO NOT PASS

Self-check / non-voting incremental assessment: the posting account is also the PR author. This ordinary comment is not an independent formal review or a merge decision.

Reviewed PR #486 targeting main, base c6678fa8df0e2ab423587eba06151e8386b5b9ce, head 5ee69128d228b921aa2446f3f0211057b22d56fa.

New blocker

[P2] Preserve the task exception reason across the real assessor subprocess boundary.

At task_protocol.py:138, the child's exception becomes a TaskProtocolError. optimization.py:212–214 then records the wrapper's class name and string rather than the original exception reason. The R3 design explicitly says the recorded reason remains "<Type>: <message>" and names the existing assessor-exception test as its guard.

I exercised the actual CLI benchmarks session start → session round path on both pins, with a synthetic task whose frozen optimization.py.assess() raises TypeError("invalid task assessment"). Persisted session state contains:

base: TypeError: invalid task assessment
head: TaskProtocolError: protocol_exception: TypeError: invalid task assessment

Both versions return exit 2, settle the attempt as infrastructure_failure, and clear pending; the regression is the recorded error-type/reason contract, not failed settlement. The existing test_assessor_exception_preserves_type_and_settles_failed_attempt still passes because it monkeypatches _session_assess to raise in the parent, bypassing the new transport.

Please retain the child exception type/message as structured data and preserve the original reason when settling a protocol_exception, while keeping transport/identity errors distinguishable. There is no need to instantiate arbitrary exception classes from child input. Add a regression that writes the throwing assessor into the task fixture before session creation, invokes the real start/round path, and checks both the returned observation and retained observation.json reason, status and cleared pending state.

Concrete follow-up to the already reported gate-test failure

The failure itself belongs to zengchuang-hw's existing review, not a new finding here. I also reproduced that exact test: base passes, head fails at line 410.

If the proposed static-isolation contract is accepted, keep _record_modules()'s expected-module assertions and the full h_entries=NEW_H_ENTRIES + records traversal. Assert zero violations, harness_model_infra == (), blockers == (), both static isolation flags true, and the exact two separately reported runtime_execution entries. Pair this with a private-copy/fixture negative control that adds a forbidden cross-core import to one of those declared record entries and must produce a violation. That checks record traversal rather than merely changing the boolean expectation. This recommendation does not establish authority to change #412's acceptance criteria.

Validation and limits

  • Nine focused CPU test files passed, 193 collected test cases: protocol child, evaluator loaders, target-module import gate, optimization session, Qwen optimization protocol, MoE benchmark, TTS benchmark/standalone, and engineering benchmark plugins. The known record-entry regression was run separately and is not included in this passing count.
  • Additional isolated probes: 20 MiB raw fd1 output returns successfully; an oversized response envelope is rejected as output_limit; a grandchild inheriting fd1 does not hold the caller open and supervision reports cleanup; SystemExit(2) is preserved; changed frozen protocol bytes are rejected as identity for both arguments and assess. These finite probes do not establish a live disk-output quota or diagnostic-retention contract; those points were already raised in the earlier review.
  • All 15 relocated TTS function/class definitions match the base AST. Gate negative tests cover a second evaluator call, another function, extra imports and a package marker. git diff --check passed; no repository source was modified.
  • Windows Job/supervision paths were traced statically, not run on native Windows. Existing session runtime-change and TTS prepared-bundle refusal messages and the documented re-prepare migration were inspected; I did not run an old-version bundle migration end to end. No full local suite, installed-wheel/platform matrix or model/device/performance experiment was run. Earlier platform/evidence concerns remain with their existing review.

Optional suggestions: none beyond the concrete test follow-up above.

Scope checked: all 23 changed files; subprocess request/response and caller failure paths; fd redirection, finite output and descendant probes; SystemExit; per-call frozen digests; Windows supervision by inspection; runtime/session identity and TTS bundle migration by inspection; MoE/TTS relocation; evaluator-loader restrictions and declared-record traversal; prior PR reviews and #412 design/conditions; isolated focused tests and exact base/head reproductions.

…l-tree gate pin (R3)

- full-check failed on both Python versions at one test:
  tests/test_campaign_record_injection.py still asserted that the real
  tree is not isolated. R3 makes it isolated, so the test now requires
  no violation, no blocker and isolation_complete. Its intent, that
  walking the extra records entries adds no violation, is unchanged.
- Review (zengchuang-hw, 5ee6912): the child's stderr reached the caller
  only for a SystemExit. task_protocol.run now copies the last 4 KiB of
  the child's stderr, protocol prints included, to the caller's stderr
  after every operation, then deletes the scratch directory. The
  constants and docs state that the size limits apply when Harness reads
  the child's files and that the deadline bounds the run. Tests assert
  the forwarded text on success, on a protocol exception and for the
  tail bound.

Refs #412

Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author)
Signed-off-by: GUOGUO <xwdong1998@163.com>
@Dong1017

Copy link
Copy Markdown
Collaborator Author

Thanks. New head: 70f6b56b7d64df99586987f80ad70c4832a37c0d. This PR had no approval, so the push voids none.

Diagnostics. I chose forwarding. After every operation, whatever its result, task_protocol.run now copies the last 4 KiB (STDERR_TAIL_BYTES) of the child's stderr to the caller's stderr, then deletes the scratch directory. Protocol prints, raw writes to descriptor 1 and grandchild output all land in that stderr. Before this commit the tail reached the caller only for a SystemExit, as you observed. test_protocol_output_goes_to_stderr_and_never_reaches_the_envelope now asserts the forwarded text for all four sources. New tests cover the forwarded text when the protocol raises, and the tail bound. Full logs are not retained after the call; the session's own records are unchanged.

Size limits. You are right that the 16 MiB check is not a live quota. The constants, CHANGELOG and docs/module-layout.md now say that both limits apply when Harness reads the child's output files, and that the 120 s deadline is what bounds the run.

CI. full-check at 5ee6912 failed on 3.10 and 3.11 with one test each (4993 passed): test_campaign_record_injection.py::test_harness_records_closure_holds_as_declared_entries still asserted not report.isolation_complete on the real tree. It is a test I missed. It now requires no violation, no blocker and isolation when the extra records entries are walked. Locally at 70f6b56: test_task_protocol_child.py, test_campaign_record_injection.py and test_target_module_import_gate.py give 79 passed and 43 subtests; Ruff 0.17.5 passes.

Acceptance provenance. The acceptance this PR targets is the current #412 body (owner revision of 2026-11-15, "Current delivery decision"): its R3 item and its final-tree conditions blockers == (), new_core_closure_isolated is True and isolation_complete is True. The design comment adds one permanent rule for the two evaluator loaders and states that it is a policy choice. The R1 neutral-leaf importer rule is the precedent. Whether that rule is accepted is an architecture decision for the issue owner. Your review and passing tests do not establish it, and this PR does not claim it. I will record that decision on #412 separately from this PR's code evidence.

The ready lane is running on 70f6b56; I will add its results to the PR body.

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent incremental review at head 70f6b56b7d64df99586987f80ad70c4832a37c0d, base c6678fa8df0e2ab423587eba06151e8386b5b9ce, compared with my reviewed head 5ee69128d228b921aa2446f3f0211057b22d56fa. Non-voting COMMENT: the changes address the earlier observations at source level; acceptance and execution verification remain incomplete. Not approval.

I inspected actual old/new source and tests, my own earlier reviews, and isolated proposed requirements. I did not read other reviewers' opinions or author result summaries.

  • Retained closure test: tests/test_campaign_record_injection.py:403–412 still supplies the record modules as additional declared Harness entries and still asserts zero violations. It now also asserts zero blockers and isolation_complete=True, instead of retaining the old false expectation. This reconciles the test with the implemented static-report model rather than removing the traversal test or weakening the gate. I also checked the unchanged gate implementation and the separate test_target_module_import_gate.py assertions for runtime_execution and Harness-to-Model/Infra paths. These remain static-closure checks, not runtime sandboxing claims. I have not freshly executed the formerly failing test on this head, so the previous regression is source-addressed, not locally verified fixed.

  • Diagnostics: src/inferrsi/harness/task_protocol.py now writes and flushes the stderr tail immediately after supervision, before transport status/envelope/result handling. The old SystemExit-only write is removed. Together with the child descriptor routing, this supports the newly explicit bounded-tail contract for ordinary success and protocol exceptions while keeping the envelope separate. Added/strengthened tests assert Python prints, raw descriptor writes, grandchild output and stderr forwarding on success; forwarding before a protocol exception; and exact tail truncation. These are meaningful assertions, not merely a successful return-value check.

  • Limits remain accurately scoped: the envelope-size check and stderr tail are read-time limits, not a live output-file quota. Scratch cleanup is via the existing TemporaryDirectory scope. The new wording does not establish filesystem/network isolation, a producer disk-write quota, or full diagnostic retention. Optional follow-up: add tail assertions to timeout/nonzero-exit/malformed-envelope paths and scratch-cleanup assertions, since the proposed contract says “every operation”; the new tests currently exercise success, protocol exception and truncation. The success test's “every byte” comment should be understood only for its small fixture, not as an unlimited-retention promise.

Current exact-head CI: ready run 44170321274 reports success for pr-references, dashboard-browser, installed-package 3.10/3.11 and Windows host compatibility. Full-check 3.10/3.11 are still in progress; macOS is queued. I am not treating these pending jobs or the prior-head results as a new full-suite pass.

Verification limits: this round is connector-based static source/test review plus CI metadata inspection. I did not execute local repository tests, Ruff, a base-versus-head replay, or model/device/service experiments: the available execution environment's isolation/trustworthiness was not established, and no independent clean test environment was available. This is a reviewer-environment limitation, not a demonstrated repository defect. The original detailed acceptance/provenance gap recorded in my earlier review remains unresolved; the proposed documentation alone does not authenticate authority to redefine that acceptance. Complete the pending exact-head checks and resolve that scope gap before treating this as approval or Issue #412 completion.

AGENTS and the review skill were checked at this head. No repository source, issue state, merge, deployment or stopped experiment was changed.

@Dong1017

Dong1017 commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator Author

Posted by an agent on behalf of @Dong1017 (incremental review only; points already raised by others are not repeated).

Incremental re-check at head 70f6b56b7d64df99586987f80ad70c4832a37c0d — still DO NOT PASS (self-check, non-voting: the posting account is also the PR author).

The previous blocker (task exception reason lost across the assessor subprocess boundary) is not addressed by this push, and the reply above does not mention it:

  • 5ee6912...70f6b56 touches task_protocol.py only for stderr forwarding and comments; task_protocol.py:143 still wraps the child's error as TaskProtocolError(kind, "<Type>: <message>").
  • optimization.py is unchanged; _failure still records type(error).__name__ of the wrapper, so the persisted reason stays TaskProtocolError: ... instead of the "<Type>: <message>" the R3 design names.

No new findings on the stderr-forwarding, size-limit wording or the test_campaign_record_injection.py pin change from a diff read.

Scope checked: read the author reply and the 5ee6912...70f6b56 compare (5 files); re-read task_protocol.py and optimization.py at the new head. Did not re-run the CLI reproduction or the test suite at 70f6b56.

…protocol path (R3)

Review follow-up (zengchuang-hw, 70f6b56): the diagnostics contract says
the child's stderr tail reaches the caller after every operation, but the
tests covered only success, a protocol exception and the tail bound. Add
the forwarded-stderr assertion to the timeout, nonzero-exit and
malformed-envelope paths, and assert that the scratch directory is gone
after a successful and a failed operation. Reword the success test's
comment so it does not promise unlimited retention. Test-only.

Refs #412

Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author)
Signed-off-by: GUOGUO <xwdong1998@163.com>

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent incremental review of 70f6b56b7d64df99586987f80ad70c4832a37c0d → 36384866a10738ff8506a4f8af42653a335aad2d, against unchanged base c6678fa8df0e2ab423587eba06151e8386b5b9ce. Non-voting COMMENT; not approval.

The incremental comparison contains only tests/test_task_protocol_child.py (30 additions, 7 deletions). I compared its actual old/new contents without reading other reviews, commit narratives or author result summaries.

The new assertions substantively address the optional coverage suggestions in my previous review:

  • Lines 152–161 check forwarded diagnostics alongside the existing nine malformed/foreign-envelope error classifications.
  • Lines 164–174 check caller stderr after the child exits nonzero without an envelope, retaining the typed exit-status assertions.
  • Lines 195–208 check a flushed diagnostic before timeout, retaining the elapsed bound and descendant late-write check.
  • Lines 211–225 direct tempfile allocation into a test-owned scratch root and assert that root is empty after success and a protocol exception. Both parameterized paths retain their result/error assertions.
  • The small-output test now correctly describes forwarding within the bounded tail, rather than promising every byte regardless of output size.

These are source-level coverage improvements, not newly executed test passes. The cleanup test covers success and protocol exceptions; its name does not establish cleanup on every transport failure, cancellation or cleanup failure. No production implementation or proposed acceptance wording changes in this increment, and I found no additional source-level blocker in this bounded test delta.

Exact-head machine evidence: ready run 44179562000 currently reports installed-package 3.10/3.11 and pr-references successful. I also inspected the completed Python 3.10 installed-package machine log: it records this head and invokes the isolated installed-wheel checker. That job is not execution evidence for these new protocol tests. Both Linux full-check jobs, Windows compatibility and dashboard-browser are still running; macOS is queued. The running full-check log is not yet available, so no new full-suite result or resolution of the earlier failing-test observation is claimed.

I did not execute repository code, tests or lint locally because no independently trusted execution environment was available. Prior results are not relabelled as this head's results. The original detailed acceptance/provenance gap and pending exact-head verification therefore still prevent approval or an Issue #412 completion claim. No source, issue state, merge or experiment was changed.

@Dong1017

Copy link
Copy Markdown
Collaborator Author

Thanks. New head: 36384866a10738ff8506a4f8af42653a335aad2d, test-only (tests/test_task_protocol_child.py). No approval existed, so the push voids none.

Retained closure test. At 70f6b56 the ready run 44170321274 has all 11 checks green, including full-check 3.10 and 3.11. That run executes test_campaign_record_injection.py::test_harness_records_closure_holds_as_declared_entries, so it is now verified on the head as well as in source. The test keeps its declared-entry traversal and zero-violation assertion and now also requires no blocker and isolation_complete. The separate runtime_execution assertions stay in test_target_module_import_gate.py.

Your optional follow-up, done. The stderr-forwarding assertion now covers the timeout, nonzero-exit and malformed-envelope paths, in addition to success, a protocol exception and the tail bound. A new test asserts that the scratch directory is gone after a successful and after a failed operation. The success test's comment now says only that each small write reaches the forwarded tail, not that output is retained without limit. test_task_protocol_child.py gives 37 passed locally; Ruff 0.17.5 passes.

Acceptance provenance. The owner's decision record is on #412 (#412 (comment)): the governing acceptance is the current issue body and its final-tree conditions, unchanged. The two evaluator loaders are accepted as the permanent final form under EVALUATOR_LOADERS. It is a policy decision and not evidence of stronger isolation, it rests on your design reply plus the owner's decision, and it lists what is not claimed and what reopens it. #486 still needs its own two approvals.

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent CI follow-up at unchanged current head 36384866a10738ff8506a4f8af42653a335aad2d, base c6678fa8df0e2ab423587eba06151e8386b5b9ce. Non-voting COMMENT: new exact-head execution evidence is now available; not approval.

Since my previous review, I checked the actual completed machine logs, not author summaries:

  • Current-head Python 3.10 full-check and Python 3.11 full-check each record this head, successful Ruff, and 4998 passed, 29 skipped, 1 warning, 671 subtests passed from pytest -rs. Neither skip summary lists the protocol test module, and the previous campaign-record failure is absent.
  • All eight jobs in current-head ready run 44179562000 now report success, including Windows/macOS compatibility, dashboard-browser, both installed-package jobs and pr-references. These other job conclusions were checked as metadata, not re-executed locally or individually promoted into broader coverage claims.
  • Separately, old-head 70f6b56b... run 44170321274 is successful. Its two Linux full-check logs each show 4996 passed, 29 skipped, 1 warning, 671 subtests passed and successful Ruff. This is historical evidence for that head, not a substitute for current-head evidence.

I reconfirmed the entire intervening delta is only tests/test_task_protocol_child.py: strengthened assertions plus the two parameterized scratch-cleanup cases, with no production changes. The +2 passing-test count is consistent with those two added cases; it is not the sole basis for attributing the result. The current-head machine runs provide the missing execution evidence for the test increment reviewed previously. The earlier full-suite regression is no longer present in these Linux runs after the retained closure assertion was reconciled with the static-report contract.

Remaining boundaries: 29 skips are still skips, including dependency-dependent image/model tests, opt-in fresh-venv checks and host-specific cases. Green targeted host jobs do not prove every skipped Linux case ran elsewhere. I did not run repository code locally, perform a new candidate-versus-base execution, or run model/device/service experiments. The original detailed acceptance/provenance gap remains unresolved, so this update clears the previously pending CI evidence, not the acceptance-authority gap or Issue #412 completion. Static isolation still is not a runtime sandbox guarantee.

No source, issue state, merge or stopped experiment was changed.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR review verdict — DO NOT PASS

Reviewed base c6678fa8df0e2ab423587eba06151e8386b5b9ce and head 36384866a10738ff8506a4f8af42653a335aad2d.

Blocker

Preserve protocol assessor exception reason across the child transport.

  1. Condition: frozen optimization.py.assess() raises TypeError("invalid task assessment") on the real child path used by benchmarks session / campaign after R3.
  2. Observed retained observation.json / CLI reason: TaskProtocolError: protocol_exception: TypeError: invalid task assessment.
  3. Expected (R3 design on #412 + pre-R3 contract in test_assessor_exception_preserves_type_and_settles_failed_attempt): TypeError: invalid task assessment.
  4. Sites: src/inferrsi/harness/task_protocol.py:143 raises TaskProtocolError(kind, "<Type>: <message>"); src/inferrsi/harness/optimization.py:212-214 _failure records type(error).__name__: {error}.
  5. Gap: tests/test_optimization_session.py:287 still monkeypatches _session_assess in-process and never exercises the child transport.
  6. Expected correction: keep child exception type/message structured; on protocol_exception settle with the original "<Type>: <message>" while leaving transport/identity failures distinguishable; add a regression that installs the throwing assessor in the task before session creation and asserts observation reason/status through real start/round.

Otherwise verified (not sufficient for PASS)

  • Five loaders closed: tip first_wave_report() empty violations/blockers/harness→model-infra; isolation flags true; Ready CI all SUCCESS on tip.
  • MoE/TTS sibling loads removed; protocol child boundary, stderr-tail forwarding, and scratch cleanup present.
  • zengchuang tip technical comments (diagnostics / gate pin / coverage) addressed at tip; #412 owner decision covers acceptance provenance. Those items are not additional blockers.

Standing steward delegation while hosted Dong-bot is on usage_limit. Acting reviewer: cursor[bot].

@Dong1017

Copy link
Copy Markdown
Collaborator Author

Posted by an authorized steward agent on behalf of @Dong1017 (hosted Dong-bot on usage_limit).

Independent formal review at tip 36384866a10738ff8506a4f8af42653a335aad2d: DO NOT PASS → Request changes by cursor[bot] (author is Dong1017, so this account cannot supply an independent Approve).

Blocker: real child-path assessor exceptions are retained as TaskProtocolError: protocol_exception: TypeError: … instead of the pre-R3 / design contract TypeError: … (task_protocol.py + optimization._failure; in-process monkeypatch test still bypasses the transport). Details: #486 (review)

Otherwise the five-loader closure and tip Ready green look solid once that reason contract is fixed + regression-covered.

…the child (R3)

Review (cursor[bot], 3638486): when a frozen optimization.py raised,
the settled attempt recorded "TaskProtocolError: protocol_exception:
TypeError: invalid task assessment" instead of "TypeError: invalid task
assessment", the reason the in-process call gave before R3 and the one
the R3 design promised. The existing test monkeypatched the assessor in
process, so it never crossed the child transport.

- TaskProtocolError for kind protocol_exception now keeps the
  protocol's own message as str() and its "<Type>: <message>" as
  `reason`. Every other kind keeps naming the transport failure, so
  transport, identity and result failures stay distinguishable.
- optimization._failure records `error.reason` for a TaskProtocolError
  and is unchanged for every other error.
- A ValueError from validate_arguments at `session start` again prints
  its own message through the CLI parser error, as before R3.
- New regressions drive the real child through `session start` and
  `round`: an assessor TypeError settles a blocked attempt with
  "TypeError: invalid task assessment" in the view and the retained
  observation.json; a validate_arguments ValueError exits 2 with its
  own message and creates no session. Protocol-child tests pin str(),
  reason and a transport failure's reason.

Refs #412

Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author)
Signed-off-by: GUOGUO <xwdong1998@163.com>
@Dong1017

Copy link
Copy Markdown
Collaborator Author

Thanks, confirmed and fixed. Before the fix, a frozen assess() raising TypeError("invalid task assessment") settled the attempt with TaskProtocolError: protocol_exception: TypeError: invalid task assessment, as you describe. That broke the R3 design's promise, and the existing test could not catch it because it monkeypatches the assessor in process.

New head: 096f93c74d63eb3870d58457a92b0403014050dd. Your CHANGES_REQUESTED review is at 3638486, and the push makes it stale; no approval existed.

  • TaskProtocolError with kind protocol_exception now keeps the protocol's own message as str() and its "<Type>: <message>" as reason. Every other kind (request, identity, result, malformed_response, invalid_response, exit_status, timeout, launch, cleanup, output_limit) still names the transport failure, so those stay distinguishable: their reason is TaskProtocolError: <kind>: ....
  • optimization._failure records error.reason for a TaskProtocolError and is unchanged for any other error.
  • The same change restores the pre-R3 CLI text at session start: a ValueError from validate_arguments prints its own message through the parser error and exits 2.

Regressions through the real child, without monkeypatching the transport:

  • test_child_assessor_exception_keeps_its_reason_through_a_real_round: the assessor raise is written into the task's optimization.py before session start, then a real round runs. The round exits 2, the session is blocked with no pending attempt, the observation's reason is TypeError: invalid task assessment, the verdict is INCONCLUSIVE with no score, and the retained observation.json matches.
  • test_child_argument_error_keeps_its_cli_message_at_start: exit 2, error: pairs must be positive on stderr without protocol_exception, and no session directory.
  • test_task_protocol_child.py pins str() and reason for a protocol exception and the reason of a transport (result) failure.

Local evidence (Windows 11, CPython 3.13.5) on the fix tree: test_optimization_session.py and test_task_protocol_child.py pass, including the two new round/start regressions. Ruff 0.17.5 and git diff --check pass. In the same run test_project_optimization.py had two failures that this change does not cause:

Neither test reaches the protocol child; the project path only gains the TaskProtocolError import in optimization.py. The ready lane on 096f93c is the full-suite proof.

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent incremental review at head 096f93c74d63eb3870d58457a92b0403014050dd, base c6678fa8df0e2ab423587eba06151e8386b5b9ce, compared with previously reviewed 36384866a10738ff8506a4f8af42653a335aad2d. Non-voting COMMENT; not approval.

I inspected the actual old/new production code and tests, caller paths, and an isolated normative view of the three added CHANGELOG lines, without reading other reviews or author result summaries.

What this increment establishes at source level

  • TaskProtocolError now keeps the protocol exception's message in str(error) and its original type/message in error.reason. Transport/identity/result failures retain their kind-labelled representation.
  • optimization._failure consumes that structured reason. A protocol exception from _session_assess therefore reaches the retained assessment without the extra transport wrapper.
  • The new real-session test exercises a child assessor exception, checks blocked/non-pending settlement, INCONCLUSIVE/no score, persisted assessment equality and the single execution log entry. The start-path test checks the original validation message, exit code 2 and absence of a created session. The additional result-serialization test distinguishes transport failure from protocol exception. These improve on a direct unit-only assertion.

One documented boundary still needs precision
The proposed statement about preserving a protocol exception's settlement reason is broader than the path just fixed. In optimization_session.py:29–45, the protocol's arguments operation runs inside the execute callback. If it raises before a receipt exists, optimization.py:344–356 retains only str(error) in coordinator_error; lines 235–236 then raise a new ValueError. By static tracing, an arguments TypeError('bad arguments') would consequently settle as ValueError: bad arguments, not TypeError: bad arguments. This is a source-derived limitation, not a locally reproduced failure. The new tests cover validate_arguments and assess, not that third operation.

Please either scope the compatibility promise explicitly to the assessor settlement/start CLI paths, or add an arguments-failure round test and preserve the structured reason through the coordinator record if all protocol-operation settlement is intended. I am not claiming this increment introduced that coordinator architecture; it limits the broader completion claim.

Current execution evidence
Ready run 44215223299 reports success for both installed-package jobs, Windows/macOS compatibility, dashboard-browser and pr-references. Both Linux full-check jobs remain in progress. The completed Windows machine log records this head and 8 passed plus 25 passed for its targeted host groups; those groups do not execute the new child-protocol/session tests, so they do not verify this increment's new assertions. Earlier-head full-suite success is not relabelled as this head's pass.

No repository code, tests or lint were executed locally because the available environment was not independently trusted. No new base/head execution or model/device/service experiment was run. The original detailed acceptance/provenance gap also remains. No source, policy, issue state, merge or stopped experiment was changed.

@Dong1017

Copy link
Copy Markdown
Collaborator Author

@Dong1017 ping

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent CI follow-up to my source review, bound to unchanged PR head 096f93c74d63eb3870d58457a92b0403014050dd and PR base c6678fa8df0e2ab423587eba06151e8386b5b9ce. Non-voting COMMENT; not approval.

Both previously pending full-check jobs have completed. I read the actual Python 3.10 log and Python 3.11 log: each reports successful Ruff and 5074 passed, 31 skipped, 1 warning, 671 subtests passed from pytest -rs. Neither skip summary lists the protocol-child or optimization-session module. All eight jobs in ready run 44215223299 now report success.

Important execution-provenance clarification: the logs check out GitHub's synthetic merge tree d384978c3f816c2624fb9b4ba7630a139475b2fe, combining this PR head with main c37832c3c7a6c457f533683b7b3386f2c3a0c004. They do not demonstrate a standalone checkout/run of the bare PR head. My earlier “exact-head” shorthand should be read as a run associated with that head, not proof of bare-head execution.

I compared that tested merge tree with the reviewed head. It contains 16 additional changed files, including three test files. The full contents of task_protocol.py, optimization.py and test_task_protocol_child.py match the reviewed head. Although test_optimization_session.py also contains main-side changes, the new start_args helper and both child-assessor/start-validation test functions match exactly. This supplies relevant integration execution evidence for the reviewed additions; the increase from the prior suite's count must not be attributed solely to this PR's three new tests.

This clears the pending CI evidence, not the source-level arguments-operation settlement boundary described in my previous review: that path still serializes the coordinator error as text and recreates a ValueError. Green tests do not establish an untested broader type-preservation contract. The original detailed acceptance/provenance gap also remains, so I am not approving or declaring Issue #412 complete.

No local repository code/tests were executed, no fresh bare-head/base comparison was run, and the 31 skipped cases are not counted as passes. No model/device experiment, source change, merge or stopped experiment was performed.

@Dong1017

Copy link
Copy Markdown
Collaborator Author

@amy-why-3459 PTAL

… record (R3)

Review (zengchuang-hw, 096f93c): when a frozen optimization.py raised in
its `arguments` operation, coordinator-error.json recorded type
"TaskProtocolError" instead of the protocol's own type, which the
in-process call recorded before R3. The settlement reason on that path
was and stays the existing coordinator behaviour, "ValueError: <message>".

- TaskProtocolError gains `type_name`: the protocol's own exception type
  for kind protocol_exception, "TaskProtocolError" for every other kind.
  It replaces the `reason` attribute added in 096f93c.
- optimization._type_name() feeds both _failure and
  coordinator-error.json, so all three operations leave the records
  they left before R3.
- New regression drives `session start` and a real `round` with an
  `arguments` TypeError: coordinator-error.json is
  {"type": "TypeError", "error": "bad arguments"}, the attempt settles
  as "ValueError: bad arguments" and the benchmark never starts.
- CHANGELOG states the record per operation instead of one broad claim.

Refs #412

Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author)
Signed-off-by: GUOGUO <xwdong1998@163.com>

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent incremental review of 096f93c74d63eb3870d58457a92b0403014050dd → bcbfbde6a593ba7d3f1f06d9b30b7f2ebfb096a8, against unchanged PR base c6678fa8df0e2ab423587eba06151e8386b5b9ce. Non-voting COMMENT; not approval.

I compared actual production/test source, the original base's coordinator path, and an isolated normative view of the CHANGELOG delta. No other reviewer opinions or author validation summaries were used.

The specific contract boundary from my previous review is now addressed at source/documentation level.

  • TaskProtocolError.type_name records the protocol's original exception type; str(error) retains its message. Other failure kinds remain labelled as TaskProtocolError.
  • optimization._type_name is used both for assessor failure formatting and for coordinator-error.json. Consequently, an arguments TypeError is retained as {"type":"TypeError","error":"bad arguments"} in that record rather than being described only as the transport wrapper.
  • The proposed CHANGELOG now separates all three operations: assess preserves the typed settlement reason; arguments preserves type/message in the coordinator record while using the existing coordinator settlement path; validate_arguments preserves the start CLI message. It no longer promises that the arguments settlement itself retains the original TypeError.
  • The new real-round test checks exactly that distinction: the typed coordinator record, ValueError: bad arguments assessment, INCONCLUSIVE/no score, and absence of the benchmark execution log. The original base already used the string-based coordinator-to-ValueError settlement path, so retaining that path is not itself a new regression.

The existing assessor/start tests and transport-failure tests are adjusted consistently for the new field. I found no new source-level blocker in this bounded increment. This is not a claim that its new test has been run successfully.

Current machine evidence: ready run 44227239166 reports both installed-package jobs, Windows compatibility and pr-references successful. Both full-check jobs and dashboard-browser remain running; macOS is queued. I inspected the completed Python 3.10 installed-package log: it invokes the isolated wheel checker on synthetic merge 57460944e10400655e4c957825640ffafe1afef2, associated with this head. That is packaging evidence, not bare-head execution or evidence that the new arguments-failure test passed.

The earlier CI results belong to earlier head/merge trees. No local repository code, tests or lint were executed because no independently trusted execution environment was available. Pending current-run verification and the original detailed acceptance/provenance gap still prevent approval or Issue #412 completion. No source, policy, issue state, merge or stopped experiment was changed.

@Dong1017

Copy link
Copy Markdown
Collaborator Author

Thanks, confirmed. Your trace is right, and checking the pre-R3 code adds one detail. Before R3 an in-process arguments TypeError("bad arguments") left coordinator-error.json as {"type": "TypeError", "error": "bad arguments"} and settled as ValueError: bad arguments, because the coordinator path re-raises the stored text as a ValueError. At 096f93c the settlement text was unchanged, but the coordinator record said TaskProtocolError. So the regression was the recorded type, and the CHANGELOG claim was broader than the code.

New head: bcbfbde6a593ba7d3f1f06d9b30b7f2ebfb096a8. No approval existed, so the push voids none.

  • TaskProtocolError now carries type_name: the protocol's own exception type for protocol_exception, and TaskProtocolError for every other kind. It replaces the reason attribute from 096f93c.
  • optimization._type_name() feeds both _failure and coordinator-error.json. All three operations now leave the records they left before R3: assess settles as "<Type>: <message>"; arguments keeps its own type and message in coordinator-error.json and settles through the unchanged coordinator path; validate_arguments prints its own message at session start and exits 2.
  • New regression test_child_arguments_exception_keeps_the_coordinator_record_through_a_real_round: the arguments raise is written into the task's optimization.py before session start, then a real round runs. The round exits 2, coordinator-error.json is {"type": "TypeError", "error": "bad arguments"}, the observation's reason is ValueError: bad arguments with an INCONCLUSIVE verdict and no score, and the benchmark never starts.
  • CHANGELOG now states the record per operation.

Local evidence (Windows 11, CPython 3.13.5) on this tree: test_optimization_session.py and test_task_protocol_child.py give 61 passed, including the three real start/round regressions; Ruff 0.17.5 and git diff --check pass. The ready lane on bcbfbde is the full-suite proof.

On provenance: the 096f93c full-check ran on GitHub's merge tree with main c37832c, as you note. I will cite it that way.

@Dong1017

Copy link
Copy Markdown
Collaborator Author

@zengchuang-hw @david6666666 can you review again?

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent CI follow-up at unchanged head bcbfbde6a593ba7d3f1f06d9b30b7f2ebfb096a8, base c6678fa8df0e2ab423587eba06151e8386b5b9ce. Non-voting COMMENT; not approval.

The pending execution evidence from my previous review is now available:

  • Actual Python 3.10 and Python 3.11 full-check logs each report successful Ruff and 5075 passed, 31 skipped, 1 warning, 671 subtests passed from pytest -rs. Neither skip summary lists the protocol-child or optimization-session module.
  • Both jobs tested synthetic merge 57460944e10400655e4c957825640ffafe1afef2, associated with this PR head, not a bare-head checkout. I compared the tested source with the reviewed head: task_protocol.py, optimization.py and test_task_protocol_child.py match completely. The optimization-session test file has other merge-side differences, but the new real-round test_child_arguments_exception_keeps_the_coordinator_record_through_a_real_round is present and matches exactly.
  • All eight jobs in ready run 44227239166 now report success. The added arguments test therefore has relevant full-suite integration evidence, rather than relying on the installed-wheel check or the passing-test count alone. The quiet logs do not provide an individual verbose PASS line for that node.

The previously identified arguments contract boundary is addressed by the source/documentation change reviewed earlier, now supported by this integration result. This update does not reopen it as an outstanding blocker. The original detailed acceptance/provenance gap remains, however, so successful CI is not sufficient for my approval or an Issue #412 completion claim.

I did not execute repository code locally, perform a fresh bare-head/base run, or run model/device experiments. The 31 skips remain unverified by these full-check runs. No source, issue state, merge or stopped experiment was changed.

cursor[bot]
cursor Bot previously approved these changes Nov 18, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base c6678fa8df0e2ab423587eba06151e8386b5b9ce and head bcbfbde6a593ba7d3f1f06d9b30b7f2ebfb096a8.
Prior DO NOT PASS at 36384866 (assessor reason wrapped as TaskProtocolError: protocol_exception: …) is resolved: protocol_exception now preserves the protocol's own type/message across the child transport, and real-round regressions assert TypeError: invalid task assessment. Local focused suite green; Ready SUCCESS on tip. Independent assist: internal/pr-486-rereview-bcbfbde6.md.

@Dong1017

Copy link
Copy Markdown
Collaborator Author

Posted by an authorized steward agent on behalf of @Dong1017 (hosted Dong-bot on usage_limit).

Independent formal re-review at tip bcbfbde6a593ba7d3f1f06d9b30b7f2ebfb096a8: PASS (prior assessor-reason CR cleared on the real child path). cursor[bot] tip Approve: #486 (review).

Author is Dong1017 — cannot self-Approve. Merge held for a write-access second Approve from an independent maintainer.

…ed task (R3)

Owner decision (#412): the gate must not name specific tasks. The two
named EVALUATOR_LOADERS entries (MoE grader, Qwen21 evaluator) tied it to
those tasks, so every new task that times submitted code would need a
gate edit.

- A task bundle program is a file directly inside
  benchmark/tasks/<bundle>/ where that directory has no __init__.py. No
  package code imports it; Harness and Apps run it as its own process.
- In a bundle program, importlib.util.module_from_spec calls are
  admitted. Name-based lookups still fail there (__import__,
  import_module, module_from_spec imported by name, a find_spec call),
  and so do static imports of other cores. The same call fails
  everywhere else.
- EVALUATOR_LOADERS and its per-function shape check are removed.
  first_wave_report().runtime_execution is computed: every bundle
  program that loads by path.
- Gate tests cover both sides: any bundle and file name is admitted,
  name-based lookups and a static Knowledge import fail, a finder class
  that only defines find_spec stays admitted, and the same call fails
  in a deeper bundle file, a bundle with __init__.py, other Benchmark
  modules, Harness, Apps and knowledge/data scripts.
- docs/module-layout.md and CHANGELOG describe the rule. No task bundle
  code changes.

Refs #412

Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author)
Signed-off-by: GUOGUO <xwdong1998@163.com>

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent incremental review of bcbfbde6a593ba7d3f1f06d9b30b7f2ebfb096a8 → 2d0fa299473c31a88bae94f9c3b9b8c59b99c559, base c6678fa8df0e2ab423587eba06151e8386b5b9ce. Non-voting COMMENT: a new static-policy gap remains; not approval.

This is a substantive policy expansion, not just cleanup: the two exact task/function/spec declarations are replaced by a location rule permitting qualified module_from_spec calls in any direct bundle file, including multiple calls and module-scope calls. The updated tests intentionally accept those cases. I assessed that proposed rule against actual source and an isolated normative documentation view, without reading other review opinions or author result claims.

1. The new find_spec prohibition misses ordinary import aliases.
tests/_gate_policy_support.py:380–397 rejects a call only when its unparsed function is exactly find_spec or importlib.util.find_spec. The subsequent name/attribute/import checks do not include find_spec. Thus this proposed fixture in benchmark/tasks/new_task/evaluate.py bypasses the documented name-lookup prohibition by static inspection:

import importlib.util
from importlib.util import find_spec as resolve
spec = resolve("inferrsi.knowledge")
module = importlib.util.module_from_spec(spec)

The ImportFrom's original name is not in DYNAMIC_IMPORT_NAMES, resolve is not in NAME_LOOKUPS, and the qualified module_from_spec call is admitted by location. import importlib.util as util; util.find_spec(...) has the same gap. These are normal import aliases, not the explicitly excluded getattr/eval/runpy mechanisms. Add negative fixtures for both forms and either resolve their bindings or conservatively reject these imported lookup uses. Keep the positive test for defining a finder method: a method definition alone is not a lookup call.

This conclusion is a direct AST-branch trace; I did not execute either fixture or claim a measured runtime exploit.

2. Do not equate “no init.py” with “cannot be imported / runs only as a separate process.”
The new predicate checks directory shape only; same-core imports remain allowed, and admitted loads remain admitted when reached through another module's closure. For example, a new identifier-named benchmark/tasks/new_task/evaluate.py can be statically imported by Benchmark package code without the predicate checking that incoming edge. Python supports namespace subpackages without init.py. The source comment's process-only/non-importable premise is therefore not established by this gate. If that separation is required, test and reject package-code imports of admitted bundle programs; otherwise describe this as a location-based static policy rather than an import/process isolation guarantee.

Evidence and remaining scope: I inspected the changed gate, both affected test files and the safe documentation contract. The unchanged evaluator behavior tests do not establish the widened allowance for every future bundle. The original detailed acceptance/provenance gap remains; the broader proposal cannot itself prove authorization to replace the previously scoped two-loader rule.

Ready run 44232306814 currently has both full-check jobs and Windows compatibility running, macOS queued, and installed-package 3.10/3.11, dashboard-browser and pr-references successful. These are job metadata, not a local test run or evidence for the missing alias fixtures. No repository code/tests/lint were executed locally because no independently trusted environment was available. No source, merge, policy setting or stopped experiment was changed.

@Dong1017

Copy link
Copy Markdown
Collaborator Author

New head: 2d0fa299473c31a88bae94f9c3b9b8c59b99c559. No approval existed, so the push voids none.

This revises Kind C after an owner decision on #412 (#412 (comment)), which replaces point 3 of the earlier record:

  • EVALUATOR_LOADERS, which named the MoE grader and the Qwen21 evaluator, is removed. The gate no longer names any task, function or call count.
  • A task bundle program is a file directly inside benchmark/tasks/<bundle>/ where that directory has no __init__.py. In a bundle program, importlib.util.module_from_spec calls are admitted. Name-based lookups still fail there (__import__, import_module, module_from_spec imported by name, a find_spec call), and so do static imports of Knowledge, Harness or Apps. The same call fails everywhere else.
  • first_wave_report().runtime_execution is now computed: every bundle program that loads by path. On the real tree it lists the same two modules as before.
  • No task bundle code changed; only tests/_gate_policy_support.py, the gate tests, docs/module-layout.md, CHANGELOG and a test docstring.

Gate tests cover both sides. Admitted: a by-path load in any bundle and file name, including a new bundle name, with one call, two calls or at module level, and a finder class that only defines find_spec, as the Qwen21 and TTS import guards do. Rejected: name-based lookups next to a by-path load, a static Knowledge import, a bundle directory given an __init__.py, a deeper file inside a bundle, and the same call in benchmark/registry.py, benchmark/tasks.py, harness/task_protocol.py, another Harness module, Apps and a knowledge/data script. The retired tables must not return.

Local (Windows 11, CPython 3.13.5): the gate, compatibility-shell, forwarder, campaign-record, first-wave policy, root-inventory and evaluator-loader tests gave 171 passed and 329 subtests. The 18 rewritten gate tests passed after I fixed one assertion: on a synthetic tree without the other cores, the report keeps their absent blockers. Ruff 0.17.5 passes. The ready lane on 2d0fa29 is the full-suite proof.

The gate cannot see which path a bundle program loads, which was equally true of the two named loaders. The rule states where by-path loading is allowed; it is not a sandbox, and no reviewer has accepted it yet.

…ograms (R3)

Review (zengchuang-hw, 2d0fa29), two static-policy gaps in the bundle
program rule:

1. find_spec was rejected only as a call spelled `find_spec(...)` or
   `importlib.util.find_spec(...)`. An alias (`from importlib.util
   import find_spec as resolve`, `import importlib.util as util;
   util.find_spec(...)`) passed. In a bundle program find_spec is now
   handled like the other dynamic-import names: any name, attribute or
   import of it fails, so aliases fail too. A finder class that only
   defines a find_spec method stays admitted.
2. "No __init__.py" did not make a bundle program unimportable: Python
   can import it through a namespace package. The gate now rejects any
   import of a bundle program from package code, Benchmark, Apps or
   another bundle, so an admitted by-path load cannot join another
   module's closure. The real tree has no such import.

The source comment, docs/module-layout.md and CHANGELOG no longer claim
the directory cannot be imported, and call the rule a location-based
static policy, not a process or sandbox guarantee. New negative
fixtures cover both alias forms, PathFinder.find_spec, module_from_spec
reached through an alias, and absolute, relative, Apps and cross-bundle
imports of an identifier-named bundle program.

Refs #412

Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author)
Signed-off-by: GUOGUO <xwdong1998@163.com>

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent incremental review of 2d0fa299473c31a88bae94f9c3b9b8c59b99c559 → 2e2de0462296bc1188a37c058f33c9ac076f7a1d, base c6678fa8df0e2ab423587eba06151e8386b5b9ce. Non-voting COMMENT, not approval.

The two concrete static-policy findings in my previous review are addressed in the changed source:

  1. Ordinary find_spec aliases: the updated scanner now includes find_spec in the bundle-specific prohibited names and checks Name, Attribute and the original ImportFrom name. This catches both previously reported alias forms without needing to resolve the alias's later call. The new fixtures also cover PathFinder.find_spec. The positive fixture defining a find_spec method remains, consistent with the AST distinction between a function definition's string name and a lookup expression.
  2. Imports of namespace-bundle programs: the report now checks incoming graph edges, independently of the ordinary same-core/Apps allowance. The five new fixtures cover absolute from-import, direct import, relative import, Apps and another bundle program. Tracing _direct_targets/_edges shows those fixtures resolve to the prohibited target. This addresses the earlier ordinary static-import counterexample; it does not establish runtime sandboxing or defeat mechanisms the scanner explicitly leaves out of scope.

I inspected the actual changed helper and regression-test source, plus an isolated normative view of the documentation delta. The new documentation's no-import and alias restrictions agree with these specific source changes. These are static conclusions, not claims that I executed the new fixtures. The source now explicitly recognizes namespace-package importability and labels the rule location-based rather than a process/sandbox guarantee.

At this check, ready run 44236885989 has successful installed-package 3.10/3.11, pr-references, dashboard-browser and Windows compatibility jobs; both full-check jobs and macOS are running. These are job-status observations, not completed full-suite evidence for the new regressions. Previous-head green runs do not prove this head passes.

The original detailed acceptance/provenance gap and the distinction between the proposed widened location rule and the formerly scoped two-loader rule remain. I therefore do not approve or claim Issue #412 complete. No repository code/tests/lint were executed locally because an independently trusted execution environment was unavailable. No source, merge, policy setting or stopped experiment was changed.

@Dong1017

Copy link
Copy Markdown
Collaborator Author

New head: 2e2de0462296bc1188a37c058f33c9ac076f7a1d. No approval existed at 2d0fa29, so this push voids none.

Both points from the review of 2d0fa29 are fixed in 2e2de04:

  1. find_spec aliases. In a bundle program, find_spec is now one of the names the dynamic-import check rejects, so any name, attribute or import of it fails. New negative fixtures next to the two earlier spellings: from importlib.util import find_spec as resolve, import importlib.util as util; util.find_spec(...) and importlib.machinery.PathFinder.find_spec(...). A finder class that only defines a find_spec method (the Qwen21 and TTS import guards) stays admitted; test_a_finder_class_in_a_bundle_program_is_not_a_lookup keeps that.
  2. "No __init__.py" does not mean "cannot be imported". Agreed: through a namespace package, an identifier-named bundle program is importable. The gate now reports <module> imports task bundle program <target> for any import edge into a bundle program. New fixtures: from inferrsi.benchmark.tasks.new_task import evaluate, import inferrsi.benchmark.tasks.new_task.evaluate and a relative import from Benchmark, plus the same import from Apps and from another bundle. The real tree has no such import, so its result is unchanged: violations == (), blockers == (), isolation_complete is True, and runtime_execution lists the MoE grader and the Qwen21 evaluator. The source comment, docs/module-layout.md, CHANGELOG and the PR body now call the rule a location-based static policy, not a process or sandbox guarantee.

The owner decision on #412 (issuecomment-6968396086) is edited to match, with a note that says what the first version got wrong.

Local evidence on the tree committed as 2e2de04 (Windows 11, CPython 3.13.5):

  • test_target_module_import_gate.py: 35 passed, 43 subtests.
  • test_first_wave_isolation_policy.py, test_knowledge_compatibility_shells.py, test_legacy_run_forwarder_gate.py, test_campaign_record_injection.py, test_package_root_inventory.py and test_evaluator_loaders.py: 147 passed, 286 subtests.
  • Ruff 0.17.5 on src tests tools and git diff --check: pass.

The ready lane on 2e2de04 is the full-suite proof.

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent CI follow-up at unchanged head 2e2de0462296bc1188a37c058f33c9ac076f7a1d, base c6678fa8df0e2ab423587eba06151e8386b5b9ce. Non-voting COMMENT, not approval.

The pending execution evidence in my previous review is now available:

  • Actual full-check logs for Python 3.10 and Python 3.11 each show successful Ruff and 5148 passed, 36 skipped, 1 warning, 671 subtests passed.
  • Both checked out the PR merge ref at 173793b1365ca4733b07afffe4f17bf2a767f470, not a bare-head tree. I fetched that tested merge's tests/_gate_policy_support.py and tests/test_target_module_import_gate.py: both match the reviewed head exactly.
  • The matching test file contains the new alias/PathFinder negative cases and all five incoming bundle-program import cases, as well as the positive finder-method definition case. Both jobs ran pytest -rs; the tested pytest configuration selects tests with quiet output, and neither skip summary lists this test module. This supplies full-suite integration evidence for the regressions rather than inferring coverage merely from a passing-test-count increase. The quiet logs do not expose an individual verbose PASS line for each parametrized case.
  • All eight jobs in ready run 44236885989 now report success.

The two prior concrete static findings remain resolved, now with this relevant integration evidence; I am not reopening them as current bugs. The original detailed acceptance/provenance gap and evidence needed to establish acceptance of the widened location-based policy remain. Green CI alone does not resolve those scope questions or establish runtime sandboxing, so this is not approval or an Issue #412 completion claim.

No local repository tests/lint, fresh bare-head/base execution or model/device experiments were run. The 36 full-check skips remain limitations of these runs; other jobs' success is reported as metadata, not as proof that every skipped case ran elsewhere. No source, merge, policy setting or stopped experiment was changed.

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR review verdict — PASS

This is a self-review by Dong-bot (authorized delegate). The PR author is Dong1017, so this is not an independent approval. The ruleset still needs two independent write-access Approves, and none exists on this head.

Reviewed base c6678fa8df0e2ab423587eba06151e8386b5b9ce and head 2e2de0462296bc1188a37c058f33c9ac076f7a1d. This head follows bcbfbde6 and 2d0fa299.

Review state on this head

  • There is no CHANGES_REQUESTED outstanding in the latest per-login decision set.
  • There is no formal APPROVED review on this exact head.
  • The only formal reviews on this head are two non-voting COMMENTs from @zengchuang-hw. The cursor[bot] review is dismissed and was for bcbfbde6, so it is not counted here.

Earlier findings

  • Assessor and arguments exception reason across the child transport. The 36384866 blocker is fixed. The fixes are 096f93c, which added reason, and bcbfbde, which replaced it with type_name. @zengchuang-hw's bcbfbde6 review confirmed this at source level. The regression runs through the real start/round path and checks the observation reason and coordinator-error.json.
  • find_spec aliases in bundle programs. The 2d0fa299 review found that aliases such as from importlib.util import find_spec as resolve were missed. @zengchuang-hw's review of 2e2de046 says this is fixed. find_spec is now rejected as a Name, an Attribute and an ImportFrom name, and the PathFinder.find_spec fixture is covered. A finder class that only defines find_spec stays admitted.
  • Imports of a bundle program through a namespace package. The same review says this is fixed. The gate now reports incoming edges into bundle programs. Five fixtures cover absolute, direct, relative, Apps and cross-bundle imports.
  • Campaign-record closure test. The 5ee6912 failure was fixed in 70f6b56, and CI has been green since.

CI evidence

  • @zengchuang-hw's CI follow-up on this head reports full-check 3.10 and 3.11 at 5148 passed, 36 skipped and 671 subtests, with Ruff successful.
  • All 8 jobs in ready run 44236885989 succeeded, per that review.
  • Those runs used the PR merge ref, not a bare-head checkout. @zengchuang-hw confirmed that tests/_gate_policy_support.py and tests/test_target_module_import_gate.py match this head exactly.

Other points

  • The knowledge-model-policy block in the PR body carries head_sha 2e2de046…, which matches the current head.
  • The PR body says the Kind C rule is an owner policy decision on #412 (issuecomment-6968396086). It also says passing tests are not a reviewer acceptance of that rule. I take that as an accurate description of what this PR claims.
  • The gate result is a static, location-based policy. It is not a sandbox or a runtime isolation guarantee, and the PR body says so.

Not verified here

  • I did not read the full diff, because none was seeded for this mention. This verdict rests on the PR body, the thread, the formal reviews and the CI evidence in them.
  • I ran no tests and did not re-run the gate.
  • No native Windows Job run, NPU or H200 comparison, or old-bundle migration was verified.

Next step: an independent maintainer Approve is still required before merge.


Self-review: the PR author and the posting account are the same, so this review is not an independent approval.

Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up to the direct review request in review 5951002790 (2026-11-18 04:53:46 UTC). Reviewed head remains 2e2de0462296bc1188a37c058f33c9ac076f7a1d, base c6678fa8df0e2ab423587eba06151e8386b5b9ce. Non-voting COMMENT, not approval.

I performed a fresh bounded check rather than treating this new request as already answered merely because the head is unchanged. The actual gate helper and regression-test file still exactly match the versions independently reviewed previously, and all eight jobs in ready run 44236885989 remain successful. The verified full-check evidence and tested-merge/source correspondence are recorded in my CI follow-up.

My technical conclusion is unchanged: the ordinary find_spec-alias and incoming bundle-program import counterexamples are addressed, with matching regression coverage in the successful integration runs. Neither is a current defect in this reviewed revision.

To preserve independent review, I used the isolated request metadata, the earlier isolated requirements, actual source and machine evidence, not another review's opinions or attributed conclusions. No additional substantive requirement could be safely separated from this new source, so I cannot claim to have adjudicated any further semantic clarification it may contain. A standalone normative statement of the intended acceptance criteria and whether the wider location-based rule replaces the earlier two-loader scope would make that remaining assessment possible.

The original detailed acceptance/provenance and widened-scope evidence gaps therefore remain; this follow-up does not establish acceptance or completion of Issue #412. No local repository execution, source changes, merge, policy changes or resumed experiments were performed.

@Dong1017

Copy link
Copy Markdown
Collaborator Author

The scope question in the reviews of 2e2de04 is now its own acceptance item in the #412 body, as an owner decision: for the evaluator by-path loads, the location-based rule replaces the earlier two-loader scope. The item supersedes point 3 of issuecomment-6961631100 (EVALUATOR_LOADERS) and the exact path/function/count form of condition 3 in issuecomment-6948387498, and calls the rule a location-based static policy, not a process or sandbox guarantee. No other line of the #412 body changed.

The head is unchanged at 2e2de0462296bc1188a37c058f33c9ac076f7a1d; no code changed.

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent requirement-change follow-up at unchanged head 2e2de0462296bc1188a37c058f33c9ac076f7a1d, base c6678fa8df0e2ab423587eba06151e8386b5b9ce. Non-voting COMMENT; no approval or Issue-completion claim.

Compared with my prior review, the normative Acceptance section of #412 now explicitly states that the location-based rule replaces the earlier exact two-loader/path/function/count scope. I verified the current Issue body and its author (Dong1017), rather than relying on the announcement or another review's interpretation.

What this resolves: the request for a standalone, unambiguous description of the intended wider static policy is now answered in the Issue body. I no longer treat the replacement wording itself as missing. This is a requirement-document change, not a new code revision or fresh test execution.

I independently re-read the current helper and regression source:

  • _is_bundle_program selects direct files under a task-bundle directory with no corresponding initializer module; it does not select an exact task/function/count list.
  • _by_path_loads admits the literal importlib.util.module_from_spec call there; other detected dynamic lookups, including ordinary find_spec aliases and PathFinder forms, remain rejected.
  • Incoming static edges to those bundle programs are rejected separately from same-core/Apps allowances. The five incoming-import fixtures and the non-package/deeper-file/other-location negative fixtures remain present.
  • The real-tree test retains empty blockers, zero Harness-to-Infra/Model paths and the separate two-entry runtime_execution report. Those two entries describe the current tree, not a hard limit on future bundle files.

Thus the code's location-based static interpretation matches the newly explicit normative text. The two previously reported concrete scanner findings remain resolved; I am not reopening them. The current-head checks remain successful. My CI evidence review records the tested merge, matching gate/test bytes and the two Linux full-check results; this round refreshed status and source, not those executions.

What remains separate: the body calls this an owner decision, but the retrieved Issue is authored by the PR author and the attributed decision is not itself independent authentication of the maintainer's acceptance of a widened policy. I can assess the explicit proposed scope without converting that attribution into new authority or clearing the earlier acceptance-provenance gap. Before approval/completion, retain a verifiable acceptance record from the authorized requirements owner for this replacement. This is not a request to repeat permission for routine reviews.

Regardless of acceptance, this scanner does not sandbox candidate code or prove runtime import/network/filesystem isolation. The body now correctly says so. No local repository code/tests/lint, bare-head/base replay, device experiment or policy-setting change was performed in this round; no other reviewers' opinions were used. There is no additional code-change demand or new optional suggestion from this requirement-only recheck.

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS for the independently reviewed R3 static-policy change.

Reviewed base c6678fa8df0e2ab423587eba06151e8386b5b9ce, exact head 2e2de0462296bc1188a37c058f33c9ac076f7a1d. Incremental range from my previous review is the same head: this round resolves the remaining requirement-source question, not a code change.

Issue-owner account confirmation now explicitly adopts the standalone location-based acceptance rule. My independent Issue follow-up records the exact scope and account-attribution limits. This closes the previous request for a direct requirements-owner acceptance record; it is not new authorization to merge or change security settings.

I re-read the unchanged gate helper and regression tests. The intended direct non-package bundle-file rule is explicit; literal by-path construction is distinguished from other detected dynamic lookups; incoming package-to-bundle-program imports are rejected. The ordinary find_spec-alias and incoming-import counterexamples from my earlier review remain addressed. No unresolved concrete code finding remains within this reviewed static-policy scope.

Validation: reopened the Python 3.10/3.11 full-check logs, each Ruff successful and 5148 passed, 36 skipped, 1 warning, 671 subtests passed. Tested merge is 173793b1365ca4733b07afffe4f17bf2a767f470; the gate/helper test content matches this head, as verified in my earlier CI review and unchanged since. Current required check records remain successful. Windows/macOS logs show their eight required host tests passing; Windows also passes the 25-case declared group. These are CI executions, not local reruns. Installed-package/browser success is check metadata, not a claim that all 36 optional skips were run elsewhere.

Approval applies to this PR's static import-policy implementation, not arbitrary runtime sandboxing, whole-Issue completion, real-model performance, external-source acceptance or the skipped optional environments. No local repository code/tests, experiment, source change or merge was performed. No other reviewers' opinions were used. No additional blocking or non-blocking change request is introduced in this requirement-only follow-up.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base c6678fa8df0e2ab423587eba06151e8386b5b9ce and head 2e2de0462296bc1188a37c058f33c9ac076f7a1d.
Prior assessor-reason blocker (36384866) remains cleared via TaskProtocolError.type_name on the real child path.
Tip delta since dismissed Approve bcbfbde6: location-based by-path policy (2d0fa299) plus find_spec-alias and incoming bundle-program import rejection (2e2de04); both zengchuang findings at 2d0fa299 cleared; #412 owner/steward acceptance of the location rule confirmed.
first_wave_report() empty violations/blockers/harness→model-infra; isolation flags true; Ready run 44236885989 SUCCESS (full-check 3.10/3.11).
No outstanding blockers.

@Dong1017
Dong1017 merged commit 91c1505 into main Nov 18, 2026
13 checks passed
@Dong1017
Dong1017 deleted the impl/r3-execution-boundaries branch November 18, 2026 08:38
@Dong1017

Copy link
Copy Markdown
Collaborator Author

Posted by an authorized steward agent on behalf of @Dong1017 (hosted Dong-bot on usage_limit).

Independent tip re-review at 2e2de0462296bc1188a37c058f33c9ac076f7a1d: PASS.

Merged (squash) → 91c150589567292292c1a37d7ccbebd7dd1015b6.

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incremental follow-up to the new mention in 6973422418.

Verified independently: the PR is merged (2026-11-18 08:38:28 UTC), with reviewed head 2e2de0462296bc1188a37c058f33c9ac076f7a1d landed as 91c150589567292292c1a37d7ccbebd7dd1015b6. The current main contains that commit. The landed gate helper and regression-test file match the reviewed bytes exactly. My existing approval remains attached to the reviewed head; no code delta or new defect in those checked files was identified.

The landed commit currently shows successful declaration/policy-manifest checks. These are policy checks, not a new product-test run. This follow-up adds landing/content-correspondence evidence only; the previous test evidence and limitations remain as recorded in my approval. The source's review links were not used as technical evidence or read for others' opinions.

No new approval is necessary. No local tests or repository code were executed, and I did not perform the merge. This confirms landing of the reviewed static-policy change, not runtime sandboxing or acceptance of the whole parent Issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:harness Harness module: profile, A/B loop, quality gates, evidence reports kind:task Bounded delivery task with a delegated slice

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants