Repository navigation
Conversation
Dong1017
left a comment
There was a problem hiding this comment.
Independent source review of head 56a40049148ced7bb80e60249a8f5df40d5c1587, against API base / observed main f93df0548a26714f6514337a8525c7c14108acad. Requesting changes for three current execution-contract defects, detailed inline: protected read/environment exposure, claim persistence after execution, and incomplete absolute-deadline enforcement.
I traced prepare/submit/serve, lane-budget transitions, confinement, leases, the CLI and all added tests against the existing architecture and #249 discussion. The attempt-scoped digest and permanently retained reservation charge are useful; they do not clear the inline execution-boundary findings. The Draft/partial-delivery description is appropriately explicit. Existing M0–M2 assignment/scope acceptance remains a maintainer decision, separate from this code review.
Verification was source and existing hosted-status inspection only. I did not execute this branch, tests, namespaces, models, devices or a paid workflow. Current hosted evidence contains the declaration check; no Ready full-check/installed-package result was observed for this head. Fixture results and author-reported tests do not qualify OS confinement or a campaign outcome. Next for the existing author @NumberWan: fix the three boundaries and supply the corresponding targeted evidence, preserving unknown/failed execution states and the Draft scope. No pilot, device use, incumbent advance, merge or Issue closure is authorized by this review.
Dong1017
left a comment
There was a problem hiding this comment.
Incremental review of the existing Draft head 56a40049: one previously unraised file-deletion handling gap in the offline submit path (inline). Existing confinement, execution-claim, and deadline findings remain in their original threads; this review does not repeat or resolve them. Scope: static inspection of parallel_round.py, its caller and fixture tests against #249; no tests, namespace checks, model calls, or device experiments were run. This is not approval or M0–M2 acceptance.
Dong1017
left a comment
There was a problem hiding this comment.
One additional blocking file-boundary finding at 56a40049: submission can copy a coordinator-readable external file through a lane-controlled symlink. This intake path is distinct from the existing candidate-execution confinement finding and must be closed before accepting untrusted lane submissions.
Verified with a disposable synthetic external canary and the exact pinned function bodies; no real private file or candidate command was accessed/executed. Existing findings remain in their threads; no duplicate findings or broader feature requirements are added.
Verification boundary
Function-level reproduction used AST-extracted prepare/submit and necessary helpers/artifact functions, standard library only, with a sanitized environment. Allowed path knobs/a.txt was replaced by a symlink to a synthetic sibling file outside work. Result: submitted=true, outside=[], source_is_symlink=true, outbox_is_regular_copy=true, copied_external_canary=true. This is not a full CLI or OS-confinement qualification; no model/hardware/network execution or repository modification occurred.
|
Addressed the five findings on 56a4004 in 9b4b08b.
Still out of scope: OS confinement on this host (user namespaces are unavailable, so the default backend fails closed), M3, incumbent advance, and shared drivers. |
Dong1017
left a comment
There was a problem hiding this comment.
Incremental static review of head 9b4b08b9 following the author's five targeted fixes: one newly identified expected-failure reporting gap in the unshare backend (inline). Existing confinement, deadline, claim and symlink findings remain on their original threads; this does not repeat, dismiss or approve them. I inspected the new confinement.py, parallel_round.py, CLI caller and available fixture tests; no real namespace, test, model or device run was performed. This is not an M0–M2 acceptance or launch authorization.
Dong1017
left a comment
There was a problem hiding this comment.
Changes requested at 9b4b08b9: the new filesystem view still lacks the runtime files needed to start a dynamically linked Python command.
- One new P2 startup defect is detailed inline. It affects the real namespace backend on an otherwise namespace-capable Linux host.
- The existing deadline follow-up and timeout-receipt finding remain applicable; this review does not repeat them.
- The persisted pre-launch claim and explicit deletion refusal improve the earlier paths. They do not establish native confinement qualification or complete the parent task. Next action remains with @NumberWan: repair the current blockers and provide bounded software evidence.
Evidence and scope
Reviewed head: 9b4b08b9f7b4bde0f18d6e3db80d8e47cf059539; API base: f93df0548a26714f6514337a8525c7c14108acad; assessed main: 919ed42c0d01dc0aa3b613085b9041d735d2d4db.
Inspected the six-file repair delta, current prepare/submit/serve and CLI consumers, runtime-root/mount/exec paths, and relevant fixture tests against the existing execution contract. Current per-head evidence contains successful declaration checks; no Ready full-check or installed-package success was observed. Verification was source and existing hosted evidence only: no repository tests, namespaces, model/device runs or paid evaluation were executed. This review did not implement the PR and grants no launch, merge or goal-acceptance authorization.
|
Addressed the three findings on 9b4b08b in fc9d232.
Still out of scope: OS-confinement qualification on this host, M3, incumbent advance, and shared drivers. |
There was a problem hiding this comment.
Reviewed commit: fc9d232
Coverage: Snapshot: 2221 files, 35993584 bytes; skipped 0 entries; Initial model context: 22 files; other accepted files available through tools; Git workspace contains a fixed snapshot, not full upstream history
| assert "SECRET_CANARY" not in plan["env"] | ||
|
|
||
|
|
||
| @pytest.mark.skipif(not namespaces_available(), reason="user namespaces are unavailable") |
There was a problem hiding this comment.
F2 — high, must-fix, test gating — the live namespace smoke fails instead of skipping when unprivileged user namespaces are blocked
- Where:
tests/test_confinement.py:135 - Problem: the guard only reads
/proc/sys/user/max_user_namespaces. When that value is positive but namespace creation is denied, the test runs and errors:UnshareBackend().run→ConfinementUnavailable: unshare: unshare failed: Operation not permitted. Executed in this container:PYTHONPATH=src python -m pytest tests/test_round_budget.py tests/test_confinement.py tests/test_parallel_round.py→1 failed, 23 passed, the failure beingtest_unshare_starts_python_and_hides_a_private_canary. Heremax_user_namespaces= 498239 andkernel.apparmor_restrict_unprivileged_userns= 1, whileunshare --user truereturns EPERM. - Conditions: unprivileged containers (default seccomp profiles) and AppArmor-restricted unprivileged-userns hosts, which is the default on Ubuntu 24.04+ and therefore on common GitHub-hosted runners. This is unrelated to whether the host has namespaces at all.
- Impact: the repository's full
pytest(ready and nightly lanes) goes red on such hosts, and the PR's statement that the smoke "skips where user namespaces are unavailable" is wrong, so this test cannot serve as the confinement qualification evidence it claims to be. - Fix: probe the syscall rather than the sysctl, e.g.
subprocess.run(["unshare", "--user", "true"], ...)and skip on a non-zero return, or wrapUnshareBackend().run(...)intry/except ConfinementUnavailableandpytest.skip(str(exc)).
There was a problem hiding this comment.
已在 cf22cb50 处理:namespaces_available() 除了读 max_user_namespaces,还实际执行一次 unshare --user --map-root-user --mount -- true,失败、超时或找不到程序都视为不可用,现场测试改为跳过。测试:test_blocked_unshare_reports_namespaces_unavailable。
|
This PR adds the offline parallel-round coordinator (prepare/submit/serve/status), lane attempt budgets, a fail-closed confinement backend and its tests. Two blocking defects: the Reviewed commit: fc9d232 |
|
Review at head
|
|
Coverage review at head
Reviewed commit range: |
|
Dong1017 在
|
|
增量 review: 已核对、确认修好的:
还需要处理的:
小问题(不阻塞): |
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Reviewed base f93df0548a26714f6514337a8525c7c14108acad and head cf22cb50f243b17227863f8e79f601b25faf5480 (full diff; no previous head was seeded). Two execution-contract blockers remain. I will re-read after the author pushes.
Blockers
-
P1 — The unshare supervisor cannot import
enter_namespaceon the layout this PR tests.UnshareBackend.run(src/inferrsi/infra/confinement.py:330) startssys.executable -c "from inferrsi.infra.confinement import enter_namespace"withenv=minimal_environment()(:346). The allowlist (:20) is onlyPATH,LANG,LC_ALL,LC_CTYPE,TZ, andPYTHONIOENCODING.PYTHONPATHis omitted, andcwdis the candidate tree, so the childsys.pathdoes not include the checkout that holdssrc/inferrsi. When that is the only way to importinferrsi, the child exits withModuleNotFoundErrorbeforeenter_namespace. Any non-zero status is raised asConfinementUnavailable(:358), andserve_ticketrecordsunsettledwithout running build or either trial.tools/parallel_round.pyserve usesselect_backend(), so a namespace-capable host takes this path.test_serve_cli_runs_without_a_caller_supplied_clocksetsPYTHONPATH=srcand acceptsunsettled, so it stays green when the jail never starts. Expected correction: put only the directory that contains theinferrsipackage (derived from this module's__file__) on the supervisor environment. Keep the candidateexecvpeenvironment on the allowlist. Add a test that the supervisor environment can importinferrsiand still omits secrets such asSECRET_CANARY. -
P2 — Restart does not treat an existing ledger row for this ticket as the same attempt.
_claimed_entry(src/inferrsi/harness/parallel_round.py:343) matches onlyphase == "claimed". Otherwiseserve_ticketalways callsadmit_attempt, which mintsordinal = len(entries)+1(:431).- Trigger A:
_save_ledgerpersistsreserved(:434) before_materialize(:437) and beforeclaimedis saved (:444).copytree/copy2raising, or a stop in that window, leaves no receipt. The next serve allocates another attempt. When the new charge exceedscap, the ticket is recordedrejectedfor budget exhaustion and never launches. When the cap has room, the same ticket runs under a second permanent charge while the first row staysreserved. - Trigger B:
_save_ledgeroffinished(:464) happens before_recordwritesresults/<ticket>.json(:465). A stop in that window has noclaimedrow, so the next serve launches build, candidate, and incumbent again.
Expected correction: a persistedrun_idending in@ticketis that attempt. Resumereservedwithout a new charge. Forclaimedorfinished, write a terminal receipt and do not launch.
- Trigger A:
Closed on this head (source inspection)
- Submit refuses symlinks via a no-follow walk, and refuses an in-glob deletion instead of hashing a missing path.
serve_ticketdrops the caller'snowand re-readsclockordatetime.now(timezone.utc)for admission and each stage. Submit checksdeadline_utc. A stageTimeoutExpiredbecomes anunsettledreceipt.claimedis saved before_launch. A laterclaimedrow with no receipt is recordedunsettledand is not launched again. Thereservedandfinishedwindows above are what remains of that contract.- The namespace plan is built from the writable set, the readable set,
runtime_roots(), andruntime_files()(PT_INTERP,lddlibraries, and/etc/ld.so.cache). Source/is refused. The coordinator environment is reduced to the allowlist.namespaces_available()probesunshare, and the live test is skipped when that probe fails. nowis optional onserve_ticket, so the CLI serve call matches the signature. A completed trial with an unreadable latency file becomesunsettled.
Follow-ups (non-blocking)
- Each stage still uses
min(seconds_remaining(...), 30)(parallel_round.py:357), so a stage stops at 30s while the lane deadline still has time left. - A symlink present at scan time is rejected.
_scanand_read_nofollowdo notfstatthe opened fd and do not open withO_NONBLOCK, so a regular file replaced by a FIFO afterstatcan block the coordinator inos.read.
Verification
Static read of parallel_round.py, round_budget.py, confinement.py, device_lease.py, tools/parallel_round.py, and the new tests, checked against the earlier threads on #268. Not verified here: pytest, unshare, namespaces, models, devices, or a paid workflow. Loader mounts and OS confinement are not qualified by this pass. Synthetic latency in this harness is not a campaign result. M0–M2 scope acceptance stays with the maintainer.
Dong-bot will re-read after the author pushes.
|
|
|
增量 review: 这次改动,已核对
这次新发现的问题
上次的问题,还没处理 以上是静态 review(diff 加 head 上的相关代码),我没有在本地跑测试。 |
|
回复 @amy-why-3459 的增量 review,新 head
本地: |
|
增量 review: 已解决,已核对
仍未关闭 以上是静态 review(diff 加 head 上的相关代码),我没有在本地跑测试。这次是校验和文档改动,不涉及精度和性能对比。 Reviewed: |
When a timed-out command's process group does not empty within the reap window, confinement raises UnreapedCommand. serve now writes quarantine/<device> for every device that ticket held, and hold_devices skips quarantined devices until the file is removed. Releasing the flock alone was not enough: the next ticket could be handed a device the old payload still used, and a one-shot serve process drops its flock on exit anyway. The _kill_group docstring now says what it checks: the process group is empty. Processes that called setsid are covered by --kill-child tearing down the PID namespace. Signed-off-by: NumberWan <wantszkin2003@gmail.com>
The confined command is PID 1 of its namespace. If it calls setsid it leaves unshare's process group, so killing the group took down unshare only; _kill_group then saw an empty group and returned while PID 1 could still hold the device node, and serve released the lease. On timeout, _kill_group now records unshare's direct children (PID 1) with their start times, kills the group and each child, and returns only once the group is empty and every recorded child is gone or a zombie. A zombie PID 1 has already outlived every other process in its namespace. If that does not happen within the reap window, UnreapedCommand is raised and serve quarantines the devices. Signed-off-by: NumberWan <wantszkin2003@gmail.com>
Quarantine now lives next to the lease file as leases/<device>.quarantined, written by DeviceLease.quarantine while the lease is still held. DeviceLease.acquire checks the marker after taking the flock and refuses the device until the file is removed, so every acquire through the lease store sees it, not only hold_devices. Signed-off-by: NumberWan <wantszkin2003@gmail.com>
_kill_group signals a recorded namespace PID 1 only if /proc still shows the same start time and it is not a zombie, so a recycled pid is left alone. _proc_stat returns None for a short /proc read instead of raising before the kill. Signed-off-by: NumberWan <wantszkin2003@gmail.com>
serve no longer has its own ABBA loop. After the confined build, a ticket is measured by run_experiment with a confined execute, and evaluate gives the verdict, as issue InferMatrixLab#249 asks (reuse run_experiment, evaluate and qualify_devices; no second runner). - The trial adapter keeps the lane deadline, command timeout, device view and read-only repository; an unreaped command still quarantines the held devices. - The receipt carries verdict, improvement_pct, reasons, medians, samples and run_dir; a ticket without a complete metric set stays unsettled. - Optional plan fields: validation (run once on the candidate), metric.min_improvement_pct (default 0), and device_model, which runs qualify_devices on the held devices before an attempt is reserved and leaves the ticket queued when the check fails. Signed-off-by: NumberWan <wantszkin2003@gmail.com>
Add close that overlays PASS-lane outboxes in lane_id order, measures the joint candidate once with run_experiment, and writes successor/ only on PASS. This is an intentional MVP, not InferMatrixLab#249 stepwise composition. Also stop host git on candidates after a writable build (_tree_identity) and map ConfinementUnavailable during validation to unsettled instead of measured FAIL, matching the review blockers on 026f686. Signed-off-by: NumberWan <wantszkin2003@gmail.com>
After rebasing onto main, RunSpec lives in inferrsi.contracts and run_experiment requires assess/report ports. Wire the v0 gates from benchmark.legacy_run_v0 and keep report as a no-op; measurement still reads evaluation.json. Signed-off-by: NumberWan <wantszkin2003@gmail.com>
2da2505 to
be9af6f
Compare
|
请再看一次 CI full-check。GPU 实机证据仍独立于这次 rebase。 |
|
be9af6f 已 rebase 到当前 main 并适配 contracts/新 run_experiment ports。请 @amy-why-3459 @Dong1017 在新 head 上再看一次;先前 head 上的 review 已因 push 变为 stale。 |
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 1acfacf8d49a13edf8d1fbdfde224c11ddbee8bf and head be9af6f579ec58c989d20d2a3d8ee08a85d98e85.
Independent re-review after rebase dismissed the prior Approve on 2da25057. Noon P1 (_tree_identity / planted-git) and P2 (validation ConfinementUnavailable → unsettled) still hold with regressions. Tip be9af6f adapts RunSpec/assess/report ports (fixes prior Ready import break). Local focused tests 58 passed / 1 skipped; ruff clean; declarations green. Ready full-check on this tip still needs to finish after fork approval — merge held for that plus a second independent LGTM. Does not close #249 / M0–M2.
|
Posted by an authorized steward agent on behalf of @Dong1017 (hosted Dong-bot on Steward re-review (afternoon): head moved off noon Noon P1/P2 still hold. Tip adapts Merge held for (1) Ready full-check green on this tip and (2) a second qualifying independent |
zengchuang-hw
left a comment
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Independent blind review of base 1acfacf and head be9af6f. Submitted as COMMENT, not REQUEST_CHANGES, under the owner's review-action constraint. No other reviewers' comments or reviews were read.
Blocking findings:
-
[P1] New files bypass the frozen protected/disjoint lane boundary (src/inferrsi/harness/parallel_round.py:313–332, 388–390). prepare checks protected intersections and differently spelled overlapping globs only against files already in the incumbent; submit checks only the submitting lane's globs. I reproduced a plan with lanes new/* and new/*.txt, protected new/protected.txt, and an incumbent with no new/ directory. prepare accepted it, then BOTH lanes successfully submitted new/protected.txt. Thus adding a file can modify a protected path and give two lanes ownership of the same path. Reject protected paths and ambiguous cross-lane ownership on each concrete submission, including newly created files; add regression coverage.
-
[P1] The namespace runtime omits the base interpreter's standard library when launched from a virtualenv (src/inferrsi/infra/confinement.py:115–123). runtime_roots includes sys.prefix (the venv) and the resolved executable's bin directory, not sys.base_prefix / the standard-library tree. On this Linux host namespaces_available() is True, but real UnshareBackend execution from a Python 3.12 venv fails at interpreter startup with ModuleNotFoundError: No module named 'encodings'. The CLI returns unsettled instead of measured. Include the necessary base runtime/stdlib locations without exposing unrelated host paths, and test actual namespace execution from a venv.
Executed:
- Read the eight-file diff, new coordinator/budget/confinement/lease implementation, CLI, affected tests and runner integration; followed AGENTS.md and review/contribution guidance, with the owner's blind-review and COMMENT restrictions taking precedence.
- Focused pytest: tests/test_parallel_round.py tests/test_round_budget.py tests/test_confinement.py: 53 passed, 5 failed, 1 skipped. Failures: serve CLI; live unshare startup/canary; both timeout-child-cleanup cases; device-node/proc/scratch test. These failures occur in the venv namespace-runtime path above; they are not claimed as five independent defects. NVIDIA/torch test skipped.
- Changed-file ruff check: passed. git diff --check: passed.
- Independent temporary-directory scope reproduction above: both forbidden submissions accepted.
- Exact-head check-run metadata includes two failures; I did not inspect their logs or infer their causes.
Coverage/acceptance limits:
This is a substantial 3,679-line, eight-file addition. Full repository pytest, accelerator execution, performance/pilot evidence, and every concurrency/security case were not validated. Original #249 requirements were supplied through an isolated requirements-only extraction, not mixed review discussion. This PR explicitly implements a narrower offline single-round slice, without the drivers/pilot and with one joint rerun rather than stepwise select-compose-rerun. Acceptance of that scope reduction remains unresolved; this review does not clear #249 or claim campaign PASS. Fix the reproduced defects, establish the accepted slice, and rerun the relevant live boundary tests before approval.
Move parallel-round assess composition into apps so harness stays clear of benchmark imports. Reject protected and cross-lane paths on concrete submit, and mount the versioned stdlib for UnshareBackend under a venv. Signed-off-by: NumberWan <wantszkin2003@gmail.com>
|
@zengchuang-hw 已在 |
zengchuang-hw
left a comment
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Independent blind incremental review of be9af6f → ceb0f1f; current target base 1acfacf. COMMENT is intentional under the owner's action constraint. No other review/comment bodies or author retellings were consulted.
Prior findings rechecked:
- The concrete protected/new-file and cross-lane ownership bypass is fixed in submit. The added regression exercises refusal for both lanes.
- The previously failing Python 3.12 virtualenv namespace path is fixed on this host. Real namespace startup, canary isolation, CLI serve and timeout cleanup tests now pass.
- Inspected the new apps assess adapter and its delegation; run_rules carries the same RunSpec fields as the previous inline conversion. Read all eight changed files in this incremental diff, including the architecture-test expectation changes.
Remaining reproduced defect:
[P2] The new no-/usr runtime test fails with a normal system interpreter (src/inferrsi/infra/confinement.py:124–143; tests/test_confinement.py:239–244). Although the new base-prefix branch excludes /usr, candidates still starts with sys.prefix. For /usr/bin/python3, sys.prefix == sys.base_prefix == /usr, so runtime_roots() returns (/usr,) and the new assertion fails. I ran this exact new pytest test with the system Python 3.13 interpreter and observed the failure. Apply the broad-root exclusion to every candidate, not just the added base prefix, while retaining the interpreter/stdlib paths needed for system Python; cover both system and venv execution. This also aligns the actual mount set with the new function's stated preference not to expose a broad /usr tree.
Validation:
- Python 3.12 venv: test_parallel_round.py, test_round_budget.py, test_confinement.py, test_knowledge_compatibility_shells.py, test_target_module_import_gate.py: 90 passed, 1 skipped, 43 subtests passed. Only NVIDIA/torch hardware case skipped.
- System Python 3.13: test_runtime_roots_include_stdlib_without_mounting_usr failed with '/usr' in roots. Reused available pure-Python pytest dependencies via PYTHONPATH; repository source was this exact head.
- Incremental changed-file ruff and git diff --check: passed.
- Exact-head check-runs lookup returned one successful check. This is not evidence that all required lanes/checks completed; logs and required-check configuration were not examined.
- Full repository suite, real accelerator/pilot and exhaustive concurrency/security validation not executed.
Acceptance remains limited to the reviewed offline slice. The earlier requirements-only extraction of #249 required stepwise composition; this code still performs a single joint rerun, and no verified accepted scope reduction has been provided. Drivers/pilot and overall #249 acceptance remain outside the demonstrated evidence. The two prior reproduced P1s are cleared for their tested conditions, but the newly failing supported-interpreter test and unresolved scope evidence prevent approval.
System interpreters use prefix=/usr; filter that broad root for all candidates and keep versioned stdlib plus the executable directory. Cover both the active interpreter and a /usr/bin/python3 subprocess. Signed-off-by: NumberWan <wantszkin2003@gmail.com>
|
@zengchuang-hw 已在 |
zengchuang-hw
left a comment
There was a problem hiding this comment.
PR review verdict — DO NOT PASS (acceptance evidence incomplete)
Independent blind incremental review of ceb0f1f → 07817ac, against target base 1acfacf. COMMENT is intentional under the owner's action constraint.
The previous /usr finding is resolved: the broad-root exclusion now applies to every candidate. I inspected the actual two-file executable-code delta through an AST view excluding comments/docstrings, and tested both interpreter configurations. No new defect was found in this incremental fix. This clears the prior P2 for the tested system and virtualenv configurations; it is not a claim of complete PR acceptance.
Executed on this exact head:
- Python 3.12 virtualenv: tests/test_parallel_round.py, test_round_budget.py, test_confinement.py, test_knowledge_compatibility_shells.py, test_target_module_import_gate.py: 91 passed, 1 skipped, 43 subtests passed.
- System /usr/bin/python3 (3.13), with available pytest dependencies supplied through PYTHONPATH: entire test_confinement.py: 20 passed, 1 skipped. This includes the formerly failing /usr assertion and actual namespace execution/startup, canary and timeout tests.
- Both skips require NVIDIA hardware and torch; no accelerator execution was performed.
- Changed-file ruff and incremental git diff --check: passed.
- AGENTS.md and the previously read review skill are unchanged in this increment.
- Exact-head check-runs lookup returned one successful check. I did not verify all required-check configuration, full repository tests, accelerator/pilot evidence or exhaustive security/concurrency behavior.
- Before submission: PR open, author NumberWan, head unchanged; no existing review by this account on this exact head.
Remaining acceptance gap, independently checked:
The isolated original requirements for #249 specify stepwise composition with a fresh remeasurement after each passing lane addition. Current close_round still materializes all selected overlays together (line 1067), calls _run_experiment once (line 1093), and then writes the successor (line 1140). The incremental fix does not change that behavior. No verified accepted reduction of that original requirement to one joint rerun is available to this review. The broader drivers/pilot deliverables are also not established by these tests. Therefore the code findings from my earlier reviews are cleared in their tested scope, but approval of this whole PR or closure of #249 would still overstate the acceptance evidence. Establish the accepted delivery slice or satisfy the original composition requirement before requesting overall approval.
No other reviewers' opinions, author comment bodies or author retellings were read.
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 1acfacf8d49a13edf8d1fbdfde224c11ddbee8bf and head 07817ac6f2c424687021f676dd3a18bbec425929 (offline prepare/submit/serve/close slice for #249; not M0–M2 closure).
Verified on tip vs afternoon be9af6f:
- Submit now refuses protected and cross-lane-owned concrete paths (including newly created files); regression
test_submit_rejects_new_protected_and_cross_lane_owned_pathspresent. runtime_rootsexcludes broad/and/usrfor every candidate while keeping versioned stdlib; venv + system-interpreter coverage added.- Assess composition moved to
apps/parallel_round.make_assess; Harness takes an injectedmake_assessport (no Benchmark import in harness). - Prior noon P1/P2 (planted
.git/ validation confinement → unsettled) still hold from earlier tips. - Local:
test_parallel_round+test_confinement59 passed / 1 skipped; ruff clean on changed paths; real-tree gate reports 21 harness→model/infra hits includingparallel_roundas this tip asserts. - Ready lane on tip was
action_required; steward approved workflow run 44045958946 (queued). Declarations success on tip.
Scope note (not a code defect): joint single-rerun close is the accepted delivery for this PR; stepwise per-lane remeasure and full #249 M0–M2 remain open. Merge held for a second current independent LGTM + Ready green on tip; gate assertions will need rebase vs R2 (#478) / current main.
Stale: findings from Oct/early-Nov heads superseded by tip 07817ac (submit protected/overlap, confinement runtime_roots, assess port). Steward re-approved current head.
|
Posted by steward agent for @Dong1017 (hosted Dong-bot on Re-review at head
@Dong1017 decide — accepted delivery slice for this PRAccept joint single-rerun Merge held for a second current independent |
zengchuang-hw
left a comment
There was a problem hiding this comment.
PR review verdict — DO NOT PASS (remaining acceptance gap)
Explicit response to the two review requests:
- 6942011059, updated 2026-11-17T07:46:50Z.
- 6943164218, updated 2026-11-17T08:44:49Z.
Current reviewed head: 07817ac; base: 1acfacf. This explicitly answers both requests rather than assuming a later review or unchanged head proves request coverage. I inspected only request metadata, my own review records and clean source/requirements, not requester opinions, author retellings or other reviews.
This pass rechecked the executable AST of submit, runtime_roots and close_round against the original isolated #249 requirements. There is no source delta since my preceding exact-head review. Concrete protected/cross-lane submission checks and the all-candidate /usr exclusion remain present. The previous findings recorded in 5934315001 and 5934770356 remain cleared for the already tested conditions.
The remaining gap is unchanged and was rechecked in code, not inferred from discussion: close_round materializes all passing overlays together at line 1067, performs one joint _run_experiment at line 1093, then writes successor at line 1140. That does not implement the original stepwise fresh-checkout remeasurement after each lane addition. No verified acceptance of the narrower delivery slice is available. Consequently I cannot approve the complete PR or clear #249 from this evidence.
Fresh checks in this pass: relevant ruff and git diff --check passed; exact-head check-runs lookup returned ten distinct checks, all successful. I did not inspect the required-check configuration or CI logs, so this is not a declaration that every required acceptance condition is complete.
Previously executed on this exact head, not rerun in this pass: five relevant test groups under Python 3.12 venv: 91 passed, 1 skipped, 43 subtests passed; entire confinement group under system Python 3.13: 20 passed, 1 skipped. Both skips require NVIDIA/torch. Full repository tests, accelerator/pilot evidence and exhaustive concurrency/security verification remain unperformed. This follow-up ran no device experiment and changed no repository code.
Both requests are now answered with this revision-bound COMMENT. That closes the response gap, not the technical/acceptance gap; COMMENT rather than REQUEST_CHANGES follows the owner's action constraint.
|
Steward note (authorized delegate for @Dong1017): tip |
|
Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5). The steward note is correct, and I'm not approving What blocks merge now
Review state
I did not re-review any code. There is no new diff to inspect yet. Re-ping @Dong1017 once the rebased head is pushed. I'll do an incremental re-check and submit a formal verdict then. (Dong-bot, authorized delegate for @Dong1017.) |
Refs #249
问题
当前 campaign 每轮只推进一个候选:从 incumbent 建一个隔离目录,改动一个声明的 plane,再与 incumbent 做 AB/BA 比较。多 Agent harness 想以集群方式在同一轮并行探索几个独立方向时,缺少以下保障:多个候选争用同一设备;工作目录不保证来自同一 incumbent;各自的改动范围可能互相覆盖;失败的尝试没有记在同一份账本里。
本 PR 的行为
本 PR 是 #249 的一个离线切片,做一轮里的
prepare/submit/serve/status/close,以及测量命令所在的隔离环境。测量按 #249 的要求复用现有 harness 的run_experiment循环,以及 v0 判定门禁(benchmark.legacy_run_v0.evaluate)和qualify_devices,不另写 runner;本 PR 注入隔离版execute,并用目录摘要做源指纹(_tree_identity,不在 build 后对候选跑 hostgit/repository_identity,避免候选里被写入的diff.external在宿主机执行)。report端口为空操作,产物以evaluation.json/events.jsonl为准,不依赖report.html。这里的「集群」指同一轮里多个 Agent 各占一条 lane、从同一个 incumbent 并行改动,测量统一交给serve。入口是python tools/parallel_round.py,不往主 CLIinferrsi加子命令。prepare)。 计划列出可用设备device_ids,并声明要比较的指标metric(名称、better: lower或higher、每次测量重复几次repeats、判定 PASS 所需的最小改进min_improvement_pct,默认 0);可选的validation是在候选上跑一次的质量检查命令,可选的device_model要求测量前检查设备型号。每条 lane 声明 id、对应的搜索计划条目、唯一的 plane(harness / model / infra)、允许修改的路径 glob、每次测量要几张设备(device_count,默认 1)、attempt 上限和截止时间。以下计划会被拒绝:两条 lane 的 glob 重叠(包括同时匹配 incumbent 中的同一文件)、lane 覆盖 protected 路径、两条 lane 对应同一条目、所有 lane 的device_count加起来超过device_ids的项数。通过后写入round-plan.json,digest 覆盖校验后的计划正文。submit)。 只比较该 lane 允许修改的文件,把差异复制到 outbox,并返回 ticket。相同 manifest 总是返回相同 ticket。这一步不占设备,集群里的 Agent 不必等测量完成。符号链接、FIFO 和其他非常规文件在读取内容前拒绝;范围内的删除在计算哈希前拒绝;改到 lane 范围外的文件、命中 protected、与其他 lane 的具体路径重叠(含新建文件),或已过截止时间,都拒绝提交。serve)。 每次调用测量一个 ticket:一次取得该 lane 所需数量的空闲设备,取不齐就全部放回,不会只占一部分。同一条 lane 的 ticket 由 lane 锁依次测量,避免同时改写同一份账本。lane 锁被占用或设备不够(包括被隔离的设备)时,ticket 保持queued,不扣 attempt;计划写了device_model时,取得设备后先用qualify_devices检查型号、以及卡上有没有其他计算进程,不通过也保持queued(收据带reason),不扣 attempt;排队时已过截止时间则记为rejected。取得设备后才 reserve,账本先记为claimed,再执行第一条命令。build(可选)在候选上跑一次;之后由run_experiment测量:incumbent 与候选的trial按 AB/BA 交替各跑repeats次,有validation时在候选上跑一次,再由evaluate给出PASS/FAIL/INCONCLUSIVE。run_experiment的产物(events.jsonl、evaluation.json等)写在<round-dir>/runs/<ticket>/,收据记下verdict、improvement_pct、reasons、每次的值、中位数和run_dir。所有命令都在隔离环境中执行;隔离不可用时拒绝执行,不回退为普通子进程。截止时间每次读取当前时钟。任何一次 trial 没有得到有效指标(失败、超时、结果文件没有计划声明的指标、或指标不是正的有限数值)时,记录unsettled,账本保持claimed。status)。 返回 ticket 的收据;还没有收据时为queued。close,joint single-rerun MVP)。 要求每个 outbox ticket 都有终态收据,且没有unsettled。把verdict==PASS的 lane 按lane_id顺序叠到同一个候选上,再对 incumbent 做一次run_experiment。联合结果为PASS时写入successor/(accepted=true);否则保留原 incumbent(retained=true)。这不是 task: parallel optimization rounds with orthogonal lanes for multi-agent harnesses #249 的逐步 composition,也不是 Claude/Codex driver 或 SwarmFlow。serve中途被杀掉后再调用同一个 ticket:账本停在reserved时沿用同一次 attempt 继续,不另扣;停在claimed或finished但没有收据时记为unsettled,不再执行;停在cancelled时记为rejected。improvement_pct一律以正数表示候选更好。verdict沿用 v0evaluate的规则:没有validation时为INCONCLUSIVE(原因写在reasons)。源完整性用目录摘要,不要求 incumbent 是 git 仓。单票收据不推进 campaign;可选的close只在本 round 目录写下successor/或明确 retain,不是 #249 的逐步合入,也不串多轮、不改调用方工作区里的「现任」路径。测试中的指标是 fixture 输入,不是 campaign 结果。Agent 自报的数字和外部 harness 的投票不作为判定依据。没有集群与有集群的控制流
阶段名沿用 dashboard 的 agent stage(
diagnosis/knowledge/search/execution/validation/review,见docs/dashboard.md)。两张图只说明一轮内的工作如何调度,不是本 PR 的性能结果。没有集群时,一轮只有一个候选。候选测量完、决策完,下一轮才开始。
有集群时,决策 Agent 在
search一次选出几个互不重叠的改法,prepare把它们冻结成几条 lane。各 lane 的 Agent 只负责在自己的范围内改出候选,交给submit,不自己测量。每条 lane 的 validation 由serve在这条 lane 自己的设备上做,不同 lane 同时进行:各 lane 的设备需求加起来不超过总设备数,所以 L1 在设备 0 上测量时,L2 可以同时在设备 1、2 上测量,互不等待。测法由计划统一规定(同一条trial命令、同一个指标、同样repeats次 AB/BA、同一个判定门槛),lane Agent 不能自己选,这样各 lane 的收据才能互相比较。review仍由决策 Agent 读收据来做。蓝色是本 PR 实现的部分;lane Agent 本身(例如 Claude Code、Codex 或脚本)由调用方启动,本 PR 不内置。几条 lane 在同一台主机上同时测量时,会共用 CPU、PCIe 和内存带宽,延迟类指标的噪声可能变大。每条 lane 的候选与 incumbent 在同一组设备上交替测量,可以抵消大部分漂移,但抵消不了 lane 之间的干扰;本 PR 没有量化这部分影响。
repeats用来压住这类噪声。单个 ticket 的状态流转
flowchart TB prepareNode[prepare 冻结轮次计划] --> submitNode[submit 提交候选并返回 ticket] submitNode --> free{lane 锁和设备都拿到了?} free -->|否,未过截止时间| queued[queued 不扣 attempt] free -->|否,已过截止时间| rejectedQ[rejected 不扣 attempt] queued --> free free -->|是,且 device_model 检查通过| deadline{reserve 后仍在截止时间前?} deadline -->|否| rejected[rejected 取消本次预留] deadline -->|是| claimed[账本记为 claimed] claimed --> runNode[隔离执行 build,再由 run_experiment AB/BA 交替跑 trial] runNode --> readable{每次 trial 都有有效指标?} readable -->|否| unsettled[unsettled 账本保持 claimed] readable -->|是| measured[measured 记录 evaluate 的 verdict、improvement_pct、每次的值与中位数]全部 ticket 终态后可另调
close(上图未展开):只合 PASS → 联合再量一次 →successor/或 retain。CLI 用法:启用并行 Agent
入口和前提
python tools/parallel_round.py <子命令>(pip install -e .后用仓库内该脚本)。主 CLIinferrsi不增加子命令;Apps 入口已在main演进,本离线切片仍留在tools/,避免绑死公共 CLI。pip install -e .。unshare --user --map-root-user --mount --pid --kill-child true能成功)。不能时,测量一律记为unsettled,不会退回普通子进程。并行几个 Agent 由什么决定
没有「开几个 Agent」的开关。计划里有几条 lane,就可以同时跑几个 Agent,每条 lane 一个。lane 数受设备限制:
device_ids:这一轮可用的设备。device_count:这条 lane 每次测量要几张设备,默认 1。例如 TP=2 的 benchmark 写 2。所有 lane 的
device_count加起来不能超过device_ids的项数,否则prepare拒绝,也就是设备用满就不再加 lane。例如 3 张设备:L1 要 1 张、L2 要 2 张,这一轮就是这两条 lane。这样每条 lane 提交后都有自己的设备可用,不用排队;代价是 Agent 在改代码时,它的设备会闲置。最小例子:两个 Agent 并行跑一轮
1. 准备 incumbent 目录(被优化的项目,
bench.py是 benchmark):2. 写轮次计划
plan.json:两条 lane,各自只能改一个文件。{ "round_id": "r1", "incumbent_id": "demo", "device_ids": ["cpu-0", "cpu-1"], "metric": {"name": "latency_ms", "better": "lower", "repeats": 3}, "protected": ["bench.py"], "trial": ["/usr/bin/python3", "bench.py", "{result}"], "lanes": [ {"id": "L1", "entry": "D-01", "plane": "infra", "globs": ["knobs/batch.txt"], "cap": 1, "per_attempt_worst_case": 1, "deadline_utc": "2026-11-13T00:00:00+00:00"}, {"id": "L2", "entry": "D-02", "plane": "harness", "globs": ["knobs/threads.txt"], "cap": 1, "per_attempt_worst_case": 1, "deadline_utc": "2026-11-13T00:00:00+00:00"} ] }3. 冻结计划,两个 Agent 各在一份 incumbent 副本里改:
4. 各自提交,再同时测量:
serve返回queued时(设备被占用),稍后对同一个 ticket 再调用即可。收据写在round-r1/results/<ticket>.json,包括verdict、improvement_pct、每次的值、中位数、run_dir和用到的设备。close写入round-r1/close.json;若联合 PASS,另有round-r1/successor/。子命令一览
prepare--spec --incumbent --round-dirsubmit--round-dir --lane --incumbent --workserve--round-dir --ticket --incumbentstatus--round-dir --ticketclose--round-dir --incumbent计划字段
device_ids{device}(build/trial/env里都可用),例如1,2。metricbetter为lower或higher;repeats(默认 1)是候选和 incumbent 各跑几次,取中位数;min_improvement_pct(默认 0)是evaluate判 PASS 的最小改进百分比。trial{result}写入{"metrics": {"<metric.name>": <正数>}}。第一个元素建议写解释器的绝对路径。validation{device};不写时 verdict 为INCONCLUSIVE。device_modelL20X。写了就在测量前对取得的设备跑qualify_devices。build{device}。command_timeout_secondsdevice_pathsshared_device_paths/runtime_paths/envprotectedlanes[]id、对应搜索计划条目entry、唯一的plane、可改路径globs、每次测量的设备数device_count(默认 1)、attempt 上限cap、单次 attempt 预算per_attempt_worst_case、截止时间deadline_utc。GPU benchmark 要多写的字段
{ "device_ids": ["0", "1", "2"], "command_timeout_seconds": 1800, "device_paths": {"0": ["/dev/nvidia0"], "1": ["/dev/nvidia1"], "2": ["/dev/nvidia2"]}, "shared_device_paths": ["/dev/nvidiactl", "/dev/nvidia-uvm", "/dev/null", "/dev/urandom"], "runtime_paths": ["/lib/x86_64-linux-gnu"], "env": {"HOME": "/tmp"} }runtime_paths写驱动用户态库(libcuda.so)所在目录,可用ldconfig -p | grep libcuda查。目前的限制
close只做 joint single-rerun(PASS 合成一次再量),不做 task: parallel optimization rounds with orthogonal lanes for multi-agent harnesses #249 的逐步 composition,也不串多轮 campaign;lane Agent 由调用方启动,不在隔离环境里。只有 build / trial /(close 时的)联合 trial 在隔离环境里执行。/proc)已实现,但还没有在真实 GPU 上验证,见下方「验证与限制」。隔离边界
unshare --pid --kill-child),并在独立 session 中启动。命令超时时杀掉unshare所在的 session,并直接杀掉 namespace 的 PID 1(PID 1 可能已经setsid离开 session);等到进程组清空、PID 1 已退出或成为 zombie 才返回,之后才放回设备。杀掉 PID 1 会拆掉整个 PID namespace,所以命令setsid出去的子进程也一并结束。serve在还持有设备的时候写入<round-dir>/leases/<device>.quarantined(内容是 ticket 和原因),然后才放回设备;只放开文件锁不够,因为单次运行的serve进程退出时锁本来就会释放。之后任何经过DeviceLease.acquire的申请都会拒绝这张设备。unsettled,原因是namespace command outlived its deadline,devices列出被隔离的设备;ls <round-dir>/leases/*.quarantined列出所有被隔离的设备。nvidia-smi看不到进程)后删除该文件,设备即可再次分配。本 PR 不自动解除。queued,过了 lane 截止时间变成rejected,不扣 attempt。这是有意的:宁可不测,也不在可能还被占用的设备上测量。/proc,从不挂宿主的/proc。每次都有空的/tmp和/dev/shm。python -I启动,包路径由参数显式传入;候选目录和PYTHON*环境变量不进入sys.path,候选里放一个同名的inferrsi包也不会被导入。PATH、locale、TZ、PYTHONIOENCODING,以及计划里的env。max_user_namespaces为 0,或unshare实际被拒绝时,后端拒绝执行。namespaces_available会实际尝试一次unshare;在这类主机上,实机测试跳过而不是失败。claimed的 attempt 不退回。例子:AURA 上的集群搜索
下面的数字不是本 PR 的代码跑出来的。AURA 的搜索用的是另一份脚本,控制流相近:同一个 incumbent 分出多条 lane,每条只改一个参数,测量结果回到同一份记录。这里只用来说明集群能省多少墙钟时间。
min_chars15),约 4.6 小时跑完全部 7 代。两种方式用的 GPU 数量和接受规则不同,以上不是同等条件下的加速比。
验证与限制
07817ac6已 rebase 到main(其后修复含 submit 边界、venv/系统 Python 的 runtime_roots、以及 apps 组装 assess),并适配inferrsi.contracts.RunSpec与新版run_experiment的assess/report端口。pytest tests/test_round_budget.py tests/test_confinement.py tests/test_parallel_round.py:rebase 后本机 focused 套件通过(含 close、目录指纹、validation 基础设施故障 →unsettled);本机没有可用的 user namespace,相关测试 skip。ruff check(本 PR 涉及文件)026f686d上的证据仍适用 confinement/lease 行为):fork run 43377269558,ubuntu-22.0453 passed,1 skipped(GPU)。ubuntu-24.04/上游ubuntu-latest因 AppArmor 会跳过 namespace 测试,真实 namespace 证据以 fork 22.04 为准。rebase 后若需可再跑一次探针。qualify_devices在 L20X 主机 GPU 0、1 上:型号匹配通过;故意填错型号时 ticket 保持queued、无账本。完整隔离测量因该机max_user_namespaces=0记unsettled。test_unshare_gpu_trial_sees_only_the_held_device与 torchprepare/submit/serve(及可选close)仍缺;需 Linux + NVIDIA + 可用 user namespace(原生 Windows 不可用)。close是否可作为 task: parallel optimization rounds with orthogonal lanes for multi-agent harnesses #249 的暂时收口实际 OS 隔离只在 GitHub 托管的 Ubuntu 22.04 + setup-python 3.11 上验证过,其他发行版或系统 Python 的加载器路径没有验证。