Skip to content

feat(harness): 離線集群 RSI 并行轮协调器 - #268

Open
NumberWan wants to merge 23 commits into
InferMatrixLab:mainfrom
NumberWan:parallel-round-offline
Open

NumberWan wants to merge 23 commits into
InferMatrixLab:mainfrom
NumberWan:parallel-round-offline

Conversation

@NumberWan

@NumberWan NumberWan commented Oct 30, 2026 •

Copy link
Copy Markdown

Refs #249

问题

当前 campaign 每轮只推进一个候选:从 incumbent 建一个隔离目录,改动一个声明的 plane,再与 incumbent 做 AB/BA 比较。多 Agent harness 想以集群方式在同一轮并行探索几个独立方向时,缺少以下保障:多个候选争用同一设备;工作目录不保证来自同一 incumbent;各自的改动范围可能互相覆盖;失败的尝试没有记在同一份账本里。

本 PR 的行为

本 PR 是 #249 的一个离线切片,做一轮里的 prepare / submit / serve / status / close,以及测量命令所在的隔离环境。测量按 #249 的要求复用现有 harness 的 run_experiment 循环,以及 v0 判定门禁(benchmark.legacy_run_v0.evaluate)和 qualify_devices,不另写 runner;本 PR 注入隔离版 execute,并用目录摘要做源指纹(_tree_identity,不在 build 后对候选跑 host git/repository_identity,避免候选里被写入的 diff.external 在宿主机执行)。report 端口为空操作,产物以 evaluation.json/events.jsonl 为准,不依赖 report.html。这里的「集群」指同一轮里多个 Agent 各占一条 lane、从同一个 incumbent 并行改动,测量统一交给 serve。入口是 python tools/parallel_round.py,不往主 CLI inferrsi 加子命令。

  1. 冻结轮次计划(prepare)。 计划列出可用设备 device_ids,并声明要比较的指标 metric(名称、better: lower 或 higher、每次测量重复几次 repeats、判定 PASS 所需的最小改进 min_improvement_pct,默认 0);可选的 validation 是在候选上跑一次的质量检查命令,可选的 device_model 要求测量前检查设备型号。每条 lane 声明 id、对应的搜索计划条目、唯一的 plane(harness / model / infra)、允许修改的路径 glob、每次测量要几张设备(device_count,默认 1)、attempt 上限和截止时间。以下计划会被拒绝:两条 lane 的 glob 重叠(包括同时匹配 incumbent 中的同一文件)、lane 覆盖 protected 路径、两条 lane 对应同一条目、所有 lane 的 device_count 加起来超过 device_ids 的项数。通过后写入 round-plan.json,digest 覆盖校验后的计划正文。
  2. 异步提交候选(submit)。 只比较该 lane 允许修改的文件,把差异复制到 outbox,并返回 ticket。相同 manifest 总是返回相同 ticket。这一步不占设备,集群里的 Agent 不必等测量完成。符号链接、FIFO 和其他非常规文件在读取内容前拒绝;范围内的删除在计算哈希前拒绝;改到 lane 范围外的文件、命中 protected、与其他 lane 的具体路径重叠(含新建文件),或已过截止时间,都拒绝提交。
  3. 多设备测量(serve)。 每次调用测量一个 ticket:一次取得该 lane 所需数量的空闲设备,取不齐就全部放回,不会只占一部分。同一条 lane 的 ticket 由 lane 锁依次测量,避免同时改写同一份账本。lane 锁被占用或设备不够(包括被隔离的设备)时,ticket 保持 queued,不扣 attempt;计划写了 device_model 时,取得设备后先用 qualify_devices 检查型号、以及卡上有没有其他计算进程,不通过也保持 queued(收据带 reason),不扣 attempt;排队时已过截止时间则记为 rejected。取得设备后才 reserve,账本先记为 claimed,再执行第一条命令。build(可选)在候选上跑一次;之后由 run_experiment 测量:incumbent 与候选的 trial 按 AB/BA 交替各跑 repeats 次,有 validation 时在候选上跑一次,再由 evaluate 给出 PASS / FAIL / INCONCLUSIVE。run_experiment 的产物(events.jsonl、evaluation.json 等)写在 <round-dir>/runs/<ticket>/,收据记下 verdict、improvement_pct、reasons、每次的值、中位数和 run_dir。所有命令都在隔离环境中执行;隔离不可用时拒绝执行,不回退为普通子进程。截止时间每次读取当前时钟。任何一次 trial 没有得到有效指标(失败、超时、结果文件没有计划声明的指标、或指标不是正的有限数值)时,记录 unsettled,账本保持 claimed。
  4. 查询结果(status)。 返回 ticket 的收据;还没有收据时为 queued。
  5. 联合收工(close,joint single-rerun MVP)。 要求每个 outbox ticket 都有终态收据,且没有 unsettled。把 verdict==PASS 的 lane 按 lane_id 顺序叠到同一个候选上,再对 incumbent 做一次 run_experiment。联合结果为 PASS 时写入 successor/(accepted=true);否则保留原 incumbent(retained=true)。这不是 task: parallel optimization rounds with orthogonal lanes for multi-agent harnesses #249 的逐步 composition,也不是 Claude/Codex driver 或 SwarmFlow。

serve 中途被杀掉后再调用同一个 ticket:账本停在 reserved 时沿用同一次 attempt 继续,不另扣;停在 claimed 或 finished 但没有收据时记为 unsettled,不再执行;停在 cancelled 时记为 rejected。

improvement_pct 一律以正数表示候选更好。verdict 沿用 v0 evaluate 的规则:没有 validation 时为 INCONCLUSIVE(原因写在 reasons)。源完整性用目录摘要,不要求 incumbent 是 git 仓。单票收据不推进 campaign;可选的 close 只在本 round 目录写下 successor/ 或明确 retain,不是 #249 的逐步合入,也不串多轮、不改调用方工作区里的「现任」路径。测试中的指标是 fixture 输入,不是 campaign 结果。Agent 自报的数字和外部 harness 的投票不作为判定依据。

没有集群与有集群的控制流

阶段名沿用 dashboard 的 agent stage(diagnosis / knowledge / search / execution / validation / review,见 docs/dashboard.md)。两张图只说明一轮内的工作如何调度,不是本 PR 的性能结果。

没有集群时,一轮只有一个候选。候选测量完、决策完,下一轮才开始。

flowchart LR
  diag[diagnosis<br/>诊断瓶颈] --> know[knowledge<br/>查经验]
  know --> search[search<br/>决策 Agent 选一个改法]
  search --> exec[execution<br/>改出一个候选]
  exec --> val[validation<br/>独占设备做 AB/BA]
  val --> review[review<br/>决策 Agent:保留或回滚]
  review -->|下一轮| diag
Loading

有集群时,决策 Agent 在 search 一次选出几个互不重叠的改法,prepare 把它们冻结成几条 lane。各 lane 的 Agent 只负责在自己的范围内改出候选,交给 submit,不自己测量。每条 lane 的 validation 由 serve 在这条 lane 自己的设备上做,不同 lane 同时进行:各 lane 的设备需求加起来不超过总设备数,所以 L1 在设备 0 上测量时,L2 可以同时在设备 1、2 上测量,互不等待。测法由计划统一规定(同一条 trial 命令、同一个指标、同样 repeats 次 AB/BA、同一个判定门槛),lane Agent 不能自己选,这样各 lane 的收据才能互相比较。review 仍由决策 Agent 读收据来做。蓝色是本 PR 实现的部分;lane Agent 本身(例如 Claude Code、Codex 或脚本)由调用方启动,本 PR 不内置。

flowchart LR
  diag[diagnosis<br/>诊断瓶颈] --> know[knowledge<br/>查经验]
  know --> plan[search<br/>决策 Agent 选 N 个互不重叠的改法<br/>prepare 冻结轮次计划]
  plan --> e1[L1 execution<br/>lane Agent 改出候选]
  plan --> e2[L2 execution<br/>lane Agent 改出候选]
  plan --> e3[L3 execution<br/>lane Agent 改出候选]
  e1 --> v1[L1 validation<br/>submit + serve 在 L1 的设备上<br/>run_experiment AB/BA + evaluate]
  e2 --> v2[L2 validation<br/>submit + serve 在 L2 的设备上<br/>run_experiment AB/BA + evaluate]
  e3 --> v3[L3 validation<br/>submit + serve 在 L3 的设备上<br/>run_experiment AB/BA + evaluate]
  v1 --> ledger[账本与收据<br/>status 查询每个 ticket]
  v2 --> ledger
  v3 --> ledger
  ledger --> closeNode[close<br/>PASS lane 合成一次再量]
  closeNode --> review[review<br/>决策 Agent 读 close/收据]
  review -->|下一轮| diag
  classDef pr fill:#dbeafe,stroke:#2563eb
  class plan,v1,v2,v3,ledger,closeNode pr
Loading

几条 lane 在同一台主机上同时测量时,会共用 CPU、PCIe 和内存带宽,延迟类指标的噪声可能变大。每条 lane 的候选与 incumbent 在同一组设备上交替测量,可以抵消大部分漂移,但抵消不了 lane 之间的干扰;本 PR 没有量化这部分影响。repeats 用来压住这类噪声。

单个 ticket 的状态流转

flowchart TB
  prepareNode[prepare 冻结轮次计划] --> submitNode[submit 提交候选并返回 ticket]
  submitNode --> free{lane 锁和设备都拿到了?}
  free -->|否,未过截止时间| queued[queued 不扣 attempt]
  free -->|否,已过截止时间| rejectedQ[rejected 不扣 attempt]
  queued --> free
  free -->|是,且 device_model 检查通过| deadline{reserve 后仍在截止时间前?}
  deadline -->|否| rejected[rejected 取消本次预留]
  deadline -->|是| claimed[账本记为 claimed]
  claimed --> runNode[隔离执行 build,再由 run_experiment AB/BA 交替跑 trial]
  runNode --> readable{每次 trial 都有有效指标?}
  readable -->|否| unsettled[unsettled 账本保持 claimed]
  readable -->|是| measured[measured 记录 evaluate 的 verdict、improvement_pct、每次的值与中位数]
Loading

全部 ticket 终态后可另调 close(上图未展开):只合 PASS → 联合再量一次 → successor/ 或 retain。

CLI 用法:启用并行 Agent

入口和前提

  • 入口是 python tools/parallel_round.py <子命令>(pip install -e . 后用仓库内该脚本)。主 CLI inferrsi 不增加子命令;Apps 入口已在 main 演进,本离线切片仍留在 tools/,避免绑死公共 CLI。
  • 安装:pip install -e .。
  • 主机要能创建 user namespace(unshare --user --map-root-user --mount --pid --kill-child true 能成功)。不能时,测量一律记为 unsettled,不会退回普通子进程。

并行几个 Agent 由什么决定

没有「开几个 Agent」的开关。计划里有几条 lane,就可以同时跑几个 Agent,每条 lane 一个。lane 数受设备限制:

  • device_ids:这一轮可用的设备。
  • 每条 lane 的 device_count:这条 lane 每次测量要几张设备,默认 1。例如 TP=2 的 benchmark 写 2。

所有 lane 的 device_count 加起来不能超过 device_ids 的项数,否则 prepare 拒绝,也就是设备用满就不再加 lane。例如 3 张设备:L1 要 1 张、L2 要 2 张,这一轮就是这两条 lane。这样每条 lane 提交后都有自己的设备可用,不用排队;代价是 Agent 在改代码时,它的设备会闲置。

最小例子:两个 Agent 并行跑一轮

1. 准备 incumbent 目录(被优化的项目,bench.py 是 benchmark):

incumbent/
  bench.py            # 读 knobs/,向 argv[1] 写 {"metrics": {"latency_ms": <数值>}}
  knobs/batch.txt
  knobs/threads.txt

2. 写轮次计划 plan.json:两条 lane,各自只能改一个文件。

{
  "round_id": "r1",
  "incumbent_id": "demo",
  "device_ids": ["cpu-0", "cpu-1"],
  "metric": {"name": "latency_ms", "better": "lower", "repeats": 3},
  "protected": ["bench.py"],
  "trial": ["/usr/bin/python3", "bench.py", "{result}"],
  "lanes": [
    {"id": "L1", "entry": "D-01", "plane": "infra", "globs": ["knobs/batch.txt"],
     "cap": 1, "per_attempt_worst_case": 1, "deadline_utc": "2026-11-13T00:00:00+00:00"},
    {"id": "L2", "entry": "D-02", "plane": "harness", "globs": ["knobs/threads.txt"],
     "cap": 1, "per_attempt_worst_case": 1, "deadline_utc": "2026-11-13T00:00:00+00:00"}
  ]
}

3. 冻结计划,两个 Agent 各在一份 incumbent 副本里改:

python tools/parallel_round.py prepare --spec plan.json --incumbent ./incumbent --round-dir ./round-r1
cp -r incumbent work-L1 && cp -r incumbent work-L2
(cd work-L1 && claude -p "只改 knobs/batch.txt,让 bench.py 的 latency_ms 更低") &
(cd work-L2 && codex exec --full-auto "只改 knobs/threads.txt,让 bench.py 的 latency_ms 更低") &
wait

4. 各自提交,再同时测量:

t1=$(python tools/parallel_round.py submit --round-dir ./round-r1 --lane L1 --incumbent ./incumbent --work ./work-L1 | jq -r .ticket)
t2=$(python tools/parallel_round.py submit --round-dir ./round-r1 --lane L2 --incumbent ./incumbent --work ./work-L2 | jq -r .ticket)
python tools/parallel_round.py serve --round-dir ./round-r1 --ticket "$t1" --incumbent ./incumbent &
python tools/parallel_round.py serve --round-dir ./round-r1 --ticket "$t2" --incumbent ./incumbent &
wait
python tools/parallel_round.py status --round-dir ./round-r1 --ticket "$t1"
python tools/parallel_round.py status --round-dir ./round-r1 --ticket "$t2"
python tools/parallel_round.py close --round-dir ./round-r1 --incumbent ./incumbent

serve 返回 queued 时(设备被占用),稍后对同一个 ticket 再调用即可。收据写在 round-r1/results/<ticket>.json,包括 verdict、improvement_pct、每次的值、中位数、run_dir 和用到的设备。close 写入 round-r1/close.json;若联合 PASS,另有 round-r1/successor/。

子命令一览

子命令 参数 作用
prepare --spec --incumbent --round-dir 检查并冻结计划
submit --round-dir --lane --incumbent --work 提交一条 lane 的改动,返回 ticket
serve --round-dir --ticket --incumbent 测量一个 ticket;不同 lane 的 ticket 可同时开多个
status --round-dir --ticket 查询 ticket 结果
close --round-dir --incumbent PASS lane 合成一次并再量一次;写 successor 或保留 incumbent

计划字段

字段 作用
device_ids 可用设备名单,名字不能含逗号。一个 ticket 取得的设备名用逗号连接后代入 {device}(build / trial / env 里都可用),例如 1,2。
metric 比较哪个指标;better 为 lower 或 higher;repeats(默认 1)是候选和 incumbent 各跑几次,取中位数;min_improvement_pct(默认 0)是 evaluate 判 PASS 的最小改进百分比。
trial 必须向 {result} 写入 {"metrics": {"<metric.name>": <正数>}}。第一个元素建议写解释器的绝对路径。
validation 可选,在候选上跑一次的质量检查,只替换 {device};不写时 verdict 为 INCONCLUSIVE。
device_model 可选,例如 L20X。写了就在测量前对取得的设备跑 qualify_devices。
build 可选,只在候选上跑一次,只替换 {device}。
command_timeout_seconds 每条 build / trial / validation 命令的上限,默认 30 秒;同时不超过 lane 的截止时间。
device_paths 每个设备 id 对应的设备节点。测量时只挂入取得的设备的节点。不写则不挂任何设备节点。
shared_device_paths / runtime_paths / env 所有测量都要的设备节点(读写)、只读的驱动库目录、额外环境变量。
protected 任何 lane 都不能改的路径,例如 benchmark 本身。
lanes[] 每条 lane 的 id、对应搜索计划条目 entry、唯一的 plane、可改路径 globs、每次测量的设备数 device_count(默认 1)、attempt 上限 cap、单次 attempt 预算 per_attempt_worst_case、截止时间 deadline_utc。

GPU benchmark 要多写的字段

{
  "device_ids": ["0", "1", "2"],
  "command_timeout_seconds": 1800,
  "device_paths": {"0": ["/dev/nvidia0"], "1": ["/dev/nvidia1"], "2": ["/dev/nvidia2"]},
  "shared_device_paths": ["/dev/nvidiactl", "/dev/nvidia-uvm", "/dev/null", "/dev/urandom"],
  "runtime_paths": ["/lib/x86_64-linux-gnu"],
  "env": {"HOME": "/tmp"}
}

runtime_paths 写驱动用户态库(libcuda.so)所在目录,可用 ldconfig -p | grep libcuda 查。

目前的限制

  • close 只做 joint single-rerun(PASS 合成一次再量),不做 task: parallel optimization rounds with orthogonal lanes for multi-agent harnesses #249 的逐步 composition,也不串多轮 campaign;lane Agent 由调用方启动,不在隔离环境里。只有 build / trial /(close 时的)联合 trial 在隔离环境里执行。
  • 设备租约和隔离标记都放在 round 目录下,只在同一个 round 内互斥;同一台主机上同时跑两个 round、用同一批设备,本 PR 不协调。
  • GPU 路径(设备节点、驱动库、新的 /proc)已实现,但还没有在真实 GPU 上验证,见下方「验证与限制」。

隔离边界

  • 新根目录只挂载可写目录、只读目录、计划声明的设备节点和驱动库、Python 运行时及动态链接器所需的库,不挂载宿主根目录。
  • 每条命令都是独立 PID namespace 里的 PID 1(unshare --pid --kill-child),并在独立 session 中启动。命令超时时杀掉 unshare 所在的 session,并直接杀掉 namespace 的 PID 1(PID 1 可能已经 setsid 离开 session);等到进程组清空、PID 1 已退出或成为 zombie 才返回,之后才放回设备。杀掉 PID 1 会拆掉整个 PID namespace,所以命令 setsid 出去的子进程也一并结束。
  • 在限定时间内确认不了这些进程已经结束时,serve 在还持有设备的时候写入 <round-dir>/leases/<device>.quarantined(内容是 ticket 和原因),然后才放回设备;只放开文件锁不够,因为单次运行的 serve 进程退出时锁本来就会释放。之后任何经过 DeviceLease.acquire 的申请都会拒绝这张设备。
    • 怎么看到: 触发隔离的那张 ticket 收据为 unsettled,原因是 namespace command outlived its deadline,devices 列出被隔离的设备;ls <round-dir>/leases/*.quarantined 列出所有被隔离的设备。
    • 怎么恢复: 由操作员确认设备上已经没有残留进程(例如 nvidia-smi 看不到进程)后删除该文件,设备即可再次分配。本 PR 不自动解除。
    • 全部设备都被隔离时: 新 ticket 一直是 queued,过了 lane 截止时间变成 rejected,不扣 attempt。这是有意的:宁可不测,也不在可能还被占用的设备上测量。
  • 只挂入本次取得的设备的节点;有设备节点时另挂一个新的 /proc,从不挂宿主的 /proc。每次都有空的 /tmp 和 /dev/shm。
  • 进入 namespace 前的 supervisor 用 python -I 启动,包路径由参数显式传入;候选目录和 PYTHON* 环境变量不进入 sys.path,候选里放一个同名的 inferrsi 包也不会被导入。
  • 子进程只继承允许的环境变量:PATH、locale、TZ、PYTHONIOENCODING,以及计划里的 env。
  • max_user_namespaces 为 0,或 unshare 实际被拒绝时,后端拒绝执行。namespaces_available 会实际尝试一次 unshare;在这类主机上,实机测试跳过而不是失败。
  • 子进程超时、隔离失败、trial 没有写出可读结果,都会留下收据;已 claimed 的 attempt 不退回。

例子:AURA 上的集群搜索

下面的数字不是本 PR 的代码跑出来的。AURA 的搜索用的是另一份脚本,控制流相近:同一个 incumbent 分出多条 lane,每条只改一个参数,测量结果回到同一份记录。这里只用来说明集群能省多少墙钟时间。

  • 目标: AURA 语音对话的首包音频延迟(TTFP)。ASR、AURA、Talker、Code2Wav 四个阶段跑在同一张 GPU 上。
  • 没有集群: 1 张 GPU 逐代搜索。约 1.8 小时找到最终保留的改动(min_chars 15),约 4.6 小时跑完全部 7 代。
  • 有集群: 6 条 lane 分到 3 张 GPU 排队测量,测量墙钟约 2.2 小时(各 lane 耗时加总约 4.1 小时)。按本 PR 的规则,同样 3 张 GPU 一轮最多 3 条单卡 lane,其余方向放到下一轮。之后改用更严格的接受规则(5 次中位数需领先 3%),复用已有测量,再用约 23 分钟确认同一个改动。
  • 同样流程搜 TTFT: 保留的配置 5 次中位数 166.7 ms,原配置 260.7 ms,约快 36%。

两种方式用的 GPU 数量和接受规则不同,以上不是同等条件下的加速比。

验证与限制

  • 当前 head 07817ac6 已 rebase 到 main(其后修复含 submit 边界、venv/系统 Python 的 runtime_roots、以及 apps 组装 assess),并适配 inferrsi.contracts.RunSpec 与新版 run_experiment 的 assess/report 端口。
  • pytest tests/test_round_budget.py tests/test_confinement.py tests/test_parallel_round.py:rebase 后本机 focused 套件通过(含 close、目录指纹、validation 基础设施故障 → unsettled);本机没有可用的 user namespace,相关测试 skip。
  • ruff check(本 PR 涉及文件)
  • 真实 user namespace(隔离语义,rebase 前 head 026f686d 上的证据仍适用 confinement/lease 行为):fork run 43377269558,ubuntu-22.04 53 passed,1 skipped(GPU)。ubuntu-24.04/上游 ubuntu-latest 因 AppArmor 会跳过 namespace 测试,真实 namespace 证据以 fork 22.04 为准。rebase 后若需可再跑一次探针。
  • qualify_devices 在 L20X 主机 GPU 0、1 上:型号匹配通过;故意填错型号时 ticket 保持 queued、无账本。完整隔离测量因该机 max_user_namespaces=0 记 unsettled。
  • GPU 实机隔离:test_unshare_gpu_trial_sees_only_the_held_device 与 torch prepare/submit/serve(及可选 close)仍缺;需 Linux + NVIDIA + 可用 user namespace(原生 Windows 不可用)。
  • reviewer 确认 attempt digest、fail-closed,以及 joint close 是否可作为 task: parallel optimization rounds with orthogonal lanes for multi-agent harnesses #249 的暂时收口

实际 OS 隔离只在 GitHub 托管的 Ubuntu 22.04 + setup-python 3.11 上验证过,其他发行版或系统 Python 的加载器路径没有验证。

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent source review of head 56a40049148ced7bb80e60249a8f5df40d5c1587, against API base / observed main f93df0548a26714f6514337a8525c7c14108acad. Requesting changes for three current execution-contract defects, detailed inline: protected read/environment exposure, claim persistence after execution, and incomplete absolute-deadline enforcement.

I traced prepare/submit/serve, lane-budget transitions, confinement, leases, the CLI and all added tests against the existing architecture and #249 discussion. The attempt-scoped digest and permanently retained reservation charge are useful; they do not clear the inline execution-boundary findings. The Draft/partial-delivery description is appropriately explicit. Existing M0–M2 assignment/scope acceptance remains a maintainer decision, separate from this code review.

Verification was source and existing hosted-status inspection only. I did not execute this branch, tests, namespaces, models, devices or a paid workflow. Current hosted evidence contains the declaration check; no Ready full-check/installed-package result was observed for this head. Fixture results and author-reported tests do not qualify OS confinement or a campaign outcome. Next for the existing author @NumberWan: fix the three boundaries and supply the corresponding targeted evidence, preserving unknown/failed execution states and the Draft scope. No pilot, device use, incumbent advance, merge or Issue closure is authorized by this review.

Comment thread src/inferrsi/infra/confinement.py Resolved
Comment thread src/inferrsi/harness/parallel_round.py Resolved
Comment thread src/inferrsi/harness/parallel_round.py Resolved

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incremental review of the existing Draft head 56a40049: one previously unraised file-deletion handling gap in the offline submit path (inline). Existing confinement, execution-claim, and deadline findings remain in their original threads; this review does not repeat or resolve them. Scope: static inspection of parallel_round.py, its caller and fixture tests against #249; no tests, namespace checks, model calls, or device experiments were run. This is not approval or M0–M2 acceptance.

Comment thread src/inferrsi/harness/parallel_round.py Resolved

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One additional blocking file-boundary finding at 56a40049: submission can copy a coordinator-readable external file through a lane-controlled symlink. This intake path is distinct from the existing candidate-execution confinement finding and must be closed before accepting untrusted lane submissions.

Verified with a disposable synthetic external canary and the exact pinned function bodies; no real private file or candidate command was accessed/executed. Existing findings remain in their threads; no duplicate findings or broader feature requirements are added.

Verification boundary

Function-level reproduction used AST-extracted prepare/submit and necessary helpers/artifact functions, standard library only, with a sanitized environment. Allowed path knobs/a.txt was replaced by a symlink to a synthetic sibling file outside work. Result: submitted=true, outside=[], source_is_symlink=true, outbox_is_regular_copy=true, copied_external_canary=true. This is not a full CLI or OS-confinement qualification; no model/hardware/network execution or repository modification occurred.

Comment thread src/inferrsi/harness/parallel_round.py Resolved
@NumberWan

Copy link
Copy Markdown
Author

Addressed the five findings on 56a4004 in 9b4b08b.

  • Filesystem view. namespace_plan builds a fresh root from the writable set, the readable set, and the Python runtime. It refuses a mount of / and does not remount the host root. The command receives an allowlisted environment (PATH, locale, TZ, PYTHONIOENCODING); the config is a coordinator tempfile, not a file under the candidate tree. FixtureBackend is still a test double (import hook plus the same minimal environment). Build and trial canaries show SECRET_CANARY and a private file stay out of both stages. Unshare failure still does not fall back to a plain process.
  • Claim before launch. serve reserves, materializes, rechecks the deadline, then advances to claimed and saves that entry before the first confined command. ConfinementUnavailable after that point leaves the entry claimed and returns unsettled. A restart with the result file removed does not reserve again and does not cancel. A refusal before launch can still cancel the reservation. The worst-case charge is kept.
  • One deadline. submit re-reads the plan and refuses an expired lane. serve rechecks at claim and before every command, and the subprocess timeout is the time left until that same deadline (capped at 30s so a far deadline does not overflow poll). Covered: reserve before expiry / launch after expiry, and a new submission after expiry, including a second read.
  • Deletions. An in-scope deletion is a structured deletion refusal before hashing. A crafted manifest with deleted is rejected before the lease, the claim, and any candidate command.
  • Symlinks. Snapshot, hash, and copy use O_NOFOLLOW from the tree root. A symlink or other non-regular entry is rejected before its target is read, including when it masks a path the incumbent still has. The external-canary fixture does not land in the outbox.

Still out of scope: OS confinement on this host (user namespaces are unavailable, so the default backend fails closed), M3, incumbent advance, and shared drivers.

@NumberWan
NumberWan marked this pull request as ready for review November 2, 2026 03:05

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incremental static review of head 9b4b08b9 following the author's five targeted fixes: one newly identified expected-failure reporting gap in the unshare backend (inline). Existing confinement, deadline, claim and symlink findings remain on their original threads; this does not repeat, dismiss or approve them. I inspected the new confinement.py, parallel_round.py, CLI caller and available fixture tests; no real namespace, test, model or device run was performed. This is not an M0–M2 acceptance or launch authorization.

Comment thread src/inferrsi/infra/confinement.py Resolved

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested at 9b4b08b9: the new filesystem view still lacks the runtime files needed to start a dynamically linked Python command.

  • One new P2 startup defect is detailed inline. It affects the real namespace backend on an otherwise namespace-capable Linux host.
  • The existing deadline follow-up and timeout-receipt finding remain applicable; this review does not repeat them.
  • The persisted pre-launch claim and explicit deletion refusal improve the earlier paths. They do not establish native confinement qualification or complete the parent task. Next action remains with @NumberWan: repair the current blockers and provide bounded software evidence.
Evidence and scope

Reviewed head: 9b4b08b9f7b4bde0f18d6e3db80d8e47cf059539; API base: f93df0548a26714f6514337a8525c7c14108acad; assessed main: 919ed42c0d01dc0aa3b613085b9041d735d2d4db.

Inspected the six-file repair delta, current prepare/submit/serve and CLI consumers, runtime-root/mount/exec paths, and relevant fixture tests against the existing execution contract. Current per-head evidence contains successful declaration checks; no Ready full-check or installed-package success was observed. Verification was source and existing hosted evidence only: no repository tests, namespaces, model/device runs or paid evaluation were executed. This review did not implement the PR and grants no launch, merge or goal-acceptance authorization.

Comment thread src/inferrsi/infra/confinement.py Resolved
@NumberWan

Copy link
Copy Markdown
Author

Addressed the three findings on 9b4b08b in fc9d232.

  • Timeout receipt. UnshareBackend turns subprocess.TimeoutExpired into ConfinementUnavailable and still deletes the coordinator config. serve catches that, and a backend that raises TimeoutExpired directly, after the claim is already durable. The receipt is unsettled, names the stage (build / candidate / incumbent), and status no longer reports queued. The claim is not cancelled.
  • Fresh deadline. The production path no longer reuses the timestamp captured when serve was entered. Each admission, claim check and command reads the clock again, and the subprocess timeout is the time left until that same absolute deadline (still capped at 30s). Covered: the clock crosses the deadline after build and before the candidate trial; the candidate command is not started, and the entry stays claimed.
  • Loader closure. The fresh root still does not mount /. Besides the Python prefix, the plan now bind-mounts the executable's PT_INTERP loader, the libraries ldd resolves, and ld.so.cache when it exists, at their original absolute paths. A private canary directory is not in that plan. The live UnshareBackend smoke is in the tree and skips where user namespaces are unavailable (max_user_namespaces=0 on this host); it was not executed here.

Still out of scope: OS-confinement qualification on this host, M3, incumbent advance, and shared drivers.

@agent-ci-infra-bot agent-ci-infra-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed commit: fc9d232
Coverage: Snapshot: 2221 files, 35993584 bytes; skipped 0 entries; Initial model context: 22 files; other accepted files available through tools; Git workspace contains a fixed snapshot, not full upstream history

Comment thread tools/parallel_round.py Resolved
Comment thread tests/test_confinement.py
assert "SECRET_CANARY" not in plan["env"]


@pytest.mark.skipif(not namespaces_available(), reason="user namespaces are unavailable")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

F2 — high, must-fix, test gating — the live namespace smoke fails instead of skipping when unprivileged user namespaces are blocked

  • Where: tests/test_confinement.py:135
  • Problem: the guard only reads /proc/sys/user/max_user_namespaces. When that value is positive but namespace creation is denied, the test runs and errors: UnshareBackend().run → ConfinementUnavailable: unshare: unshare failed: Operation not permitted. Executed in this container: PYTHONPATH=src python -m pytest tests/test_round_budget.py tests/test_confinement.py tests/test_parallel_round.py → 1 failed, 23 passed, the failure being test_unshare_starts_python_and_hides_a_private_canary. Here max_user_namespaces = 498239 and kernel.apparmor_restrict_unprivileged_userns = 1, while unshare --user true returns EPERM.
  • Conditions: unprivileged containers (default seccomp profiles) and AppArmor-restricted unprivileged-userns hosts, which is the default on Ubuntu 24.04+ and therefore on common GitHub-hosted runners. This is unrelated to whether the host has namespaces at all.
  • Impact: the repository's full pytest (ready and nightly lanes) goes red on such hosts, and the PR's statement that the smoke "skips where user namespaces are unavailable" is wrong, so this test cannot serve as the confinement qualification evidence it claims to be.
  • Fix: probe the syscall rather than the sysctl, e.g. subprocess.run(["unshare", "--user", "true"], ...) and skip on a non-zero return, or wrap UnshareBackend().run(...) in try/except ConfinementUnavailable and pytest.skip(str(exc)).

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已在 cf22cb50 处理:namespaces_available() 除了读 max_user_namespaces,还实际执行一次 unshare --user --map-root-user --mount -- true,失败、超时或找不到程序都视为不可用,现场测试改为跳过。测试:test_blocked_unshare_reports_namespaces_unavailable。

Comment thread src/inferrsi/harness/parallel_round.py Resolved
@agent-ci-infra-bot

Copy link
Copy Markdown

This PR adds the offline parallel-round coordinator (prepare/submit/serve/status), lane attempt budgets, a fail-closed confinement backend and its tests. Two blocking defects: the serve CLI now calls serve_ticket() without its still-required now argument (TypeError on the only production entry point), and the new live namespace test errors instead of skipping when unprivileged user namespaces are blocked. Executed here: PYTHONPATH=src pytest tests/test_round_budget.py tests/test_confinement.py tests/test_parallel_round.py → 1 failed, 23 passed; ruff could not be executed in this container (binary not runnable), and no test-lane CI check runs exist for this head (only the knowledge-policy check).
Earlier findings: F(56a4004 #1 read/credential boundary) fixed — the fresh root excludes /, readable is mounted, and only an allowlisted env reaches the command; F(#2 claim before launch) fixed — claimed is persisted before the first command; F(#3 deadline) fixed in the engine, but the fix broke the CLI wiring (F1); F(#4 deletion) fixed — structured deletion refusal before hashing; F(#5 symlink) fixed — no-follow snapshot/read with linked-path rejection. F(9b4b08b timeout receipt) fixed — TimeoutExpired becomes ConfinementUnavailable/unsettled receipt. F(9b4b08b clock) fixed — default current() reads a fresh clock. F(9b4b08b loader closure) fixed in code, but the added smoke could not be validated here (unprivileged unshare returns EPERM) and mis-gates (F2).
Labels: kind:task (bounded delivery slice of the #249 coordinator work; no priority label returned because the repository requires a status decision for priority:*).

Reviewed commit: fc9d232
Coverage: Snapshot: 2221 files, 35993584 bytes; skipped 0 entries; Initial model context: 22 files; other accepted files available through tools; Git workspace contains a fixed snapshot, not full upstream history

@agent-ci-infra-bot agent-ci-infra-bot Bot added kind:task Bounded delivery task with a delegated slice bot-reviewed labels Nov 2, 2026

Copy link
Copy Markdown
Collaborator

Review at head fc9d2328: not approving yet.

  1. serve CLI is broken (tools/parallel_round.py:53-58): serve_ticket(round_dir, ticket, incumbent, backend=select_backend()) omits now, but src/inferrsi/harness/parallel_round.py:388-396 still declares now: datetime as a required keyword-only argument, so the only production entry point raises TypeError. Expected: pass now=datetime.now(timezone.utc) (or make now optional and derive it from clock) and add a CLI-level test.
  2. Open bot findings at this head: F2 (tests/test_confinement.py:135, live namespace smoke fails instead of skipping when user namespaces are blocked) and F3 (src/inferrsi/harness/parallel_round.py:446, a trial exiting 0 without a valid result leaves no terminal receipt) have no reply or fix.
  3. Open CHANGES_REQUESTED from Dong1017 (runtime dependency closure in the fresh root, src/inferrsi/infra/confinement.py): please confirm whether fc9d2328 addresses it and reply in that thread.
  4. Ready CI has not run at this head (only declarations).

Copy link
Copy Markdown
Collaborator

Coverage review at head fc9d2328a6ef1a9b79e98b961967699622f5e305 (base f93df0548a26714f6514337a8525c7c14108acad). Re-checked against my earlier short-SHA note and the bot review at this head. No new commits. Not approving yet.

  1. serve CLI is still broken (tools/parallel_round.py ~L53–58): serve_ticket(round_dir, ticket, incumbent, backend=select_backend()) omits now, but src/inferrsi/harness/parallel_round.py ~L388–396 still declares now: datetime as a required keyword-only argument, so the only production entry point raises TypeError. (Inside the function the body uses current() / clock and does not read now, so making now optional — or removing it and passing now=datetime.now(timezone.utc) from the CLI — both work; please pick one and add a CLI-level test.)
  2. Open bot findings at this head (agent-ci-infra-bot): F2 (tests/test_confinement.py live namespace smoke fails instead of skipping when user namespaces are blocked) and F3 (src/inferrsi/harness/parallel_round.py ~L446, a trial exiting 0 without a valid result leaves no terminal receipt) have no reply or fix on this head.
  3. Outstanding Dong1017 CHANGES_REQUESTED (runtime dependency closure in the fresh root, src/inferrsi/infra/confinement.py, and earlier threads): author claims on fc9d232 address loader closure via PT_INTERP / ldd / ld.so.cache, but please confirm in those review threads whether the blockers are cleared and supply the targeted evidence.
  4. Ready CI has not run at this head (only declaration-related evidence observed previously).

Reviewed commit range: f93df0548a26714f6514337a8525c7c14108acad … fc9d2328a6ef1a9b79e98b961967699622f5e305.

@NumberWan

Copy link
Copy Markdown
Author

fc9d232 上的三点已经在 cf22cb5 处理。说明改成简体中文。

  1. serve CLI。 serve_ticket 的 now 改成可选,生产入口不再因为少传这个参数而 TypeError。函数内部仍用 clock 或当前时间,不把进场时间冻成截止时间。补了 test_serve_cli_runs_without_a_caller_supplied_clock。
  2. namespace 测试误失败。 namespaces_available 在 max_user_namespaces > 0 之后再跑一次 unshare --user --map-root-user --mount true。返回非 0、超时或找不到命令时视为不可用,现场测试走 skip。补了 test_blocked_unshare_reports_namespaces_unavailable。这台机器 max_user_namespaces 是 0,原有的现场测试仍然 skipped,没有实际跑进 namespace。
  3. 没有结果文件就没有收据。 build / candidate / incumbent 都返回 completed,但 latency JSON 缺失、损坏或缺字段时,写下 unsettled,原因写明 trial 没有可读的 latency 结果。claim 不取消。补了 test_completed_trial_without_a_latency_file_is_unsettled。

Dong1017 在 9b4b08b 上提的动态链接器闭合,仍是 fc9d232 里用 PT_INTERP、ldd 和 ld.so.cache 做的那一版;这次没有改那条挂载逻辑。这台机器不能创建 user namespace,所以那条现场启动测试这里仍然跳过。

pytest tests/test_round_budget.py tests/test_confinement.py tests/test_parallel_round.py:26 passed,1 skipped。ruff check 通过。PR 仍不把 M0–M2 当成已验收。

Copy link
Copy Markdown
Collaborator

增量 review:fc9d2328a6ef1a9b79e98b961967699622f5e305..cf22cb50f243b17227863f8e79f601b25faf5480(1 个提交,base f93df0548a26714f6514337a8525c7c14108acad)。谢谢这么快修完,这次先不 approve。

已核对、确认修好的:

  • serve CLI:src/inferrsi/harness/parallel_round.py L394 now 改为可选并 del now,tools/parallel_round.py L53-58 不传 now 也不会再 TypeError;截止时间仍由 current() 每次重新读时钟。
  • F2 namespace 误失败:src/inferrsi/infra/confinement.py namespaces_available() 现在实际跑一次 unshare --user --map-root-user --mount -- true,非 0 / 超时 / OSError 都返回 False,test_blocked_unshare_reports_namespaces_unavailable 覆盖了 EPERM 情形。
  • F3 缺结果无收据:parallel_round.py L451-462 对读取 / 解析 / 缺字段统一写 unsettled,且在 advance(..., "finished") 之前返回,账本停在 claimed,与之前的约定一致。

还需要处理的:

  1. Ready CI 没在这个 head 上跑:cf22cb50 目前只有 declarations,没有 full-check (3.10/3.11)、installed-package、pr-references。PR 描述里的 26 passed / 1 skipped 是本地结果,请触发完整 CI,让新增的 CLI 子进程测试在 CI 环境里也跑一遍。
  2. tests/test_parallel_round.py L409-439 的 CLI 测试断言太宽:body["status"] in {"measured", "unsettled", "rejected", "queued"} 几乎接受任何结果,只能证明没有 TypeError。计划截止时间是 2099 年,所以这里会走真实 select_backend():没有 user namespace 时应该确定是 unsettled(UnavailableBackend)。建议按 namespaces_available() 分支断言具体状态,至少在不可用时断言 unsettled 且 reason 指向 confinement 不可用,这样以后 fail-closed 行为退化时测试能抓到。
  3. Dong1017 的 CHANGES_REQUESTED 仍未关闭:9b4b08b9 上关于 fresh root 动态链接器闭合的 thread(src/inferrsi/infra/confinement.py L99)以及更早几条 thread 都还没有作者回复。这次没改挂载逻辑,现场 UnshareBackend 启动测试在你和 bot 的机器上都被跳过,所以 PT_INTERP / ldd / ld.so.cache 这一版在真实 namespace 里能否启动 Python 还没有人验证过。请在对应 thread 里回复,最好能在一台能创建 user namespace 的机器(或 CI runner)上跑一次 test_unshare_starts_python_and_hides_a_private_canary 并贴出结果,再请 Dong1017 确认。

小问题(不阻塞):namespaces_available() 每次调用都会起一个 unshare 子进程,select_backend() 只在 serve 入口调一次所以问题不大;如果以后在循环里调用,可以考虑缓存结果。

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR review verdict — DO NOT PASS

Reviewed base f93df0548a26714f6514337a8525c7c14108acad and head cf22cb50f243b17227863f8e79f601b25faf5480 (full diff; no previous head was seeded). Two execution-contract blockers remain. I will re-read after the author pushes.

Blockers

  1. P1 — The unshare supervisor cannot import enter_namespace on the layout this PR tests. UnshareBackend.run (src/inferrsi/infra/confinement.py:330) starts sys.executable -c "from inferrsi.infra.confinement import enter_namespace" with env=minimal_environment() (:346). The allowlist (:20) is only PATH, LANG, LC_ALL, LC_CTYPE, TZ, and PYTHONIOENCODING. PYTHONPATH is omitted, and cwd is the candidate tree, so the child sys.path does not include the checkout that holds src/inferrsi. When that is the only way to import inferrsi, the child exits with ModuleNotFoundError before enter_namespace. Any non-zero status is raised as ConfinementUnavailable (:358), and serve_ticket records unsettled without running build or either trial. tools/parallel_round.py serve uses select_backend(), so a namespace-capable host takes this path. test_serve_cli_runs_without_a_caller_supplied_clock sets PYTHONPATH=src and accepts unsettled, so it stays green when the jail never starts. Expected correction: put only the directory that contains the inferrsi package (derived from this module's __file__) on the supervisor environment. Keep the candidate execvpe environment on the allowlist. Add a test that the supervisor environment can import inferrsi and still omits secrets such as SECRET_CANARY.

  2. P2 — Restart does not treat an existing ledger row for this ticket as the same attempt. _claimed_entry (src/inferrsi/harness/parallel_round.py:343) matches only phase == "claimed". Otherwise serve_ticket always calls admit_attempt, which mints ordinal = len(entries)+1 (:431).

    • Trigger A: _save_ledger persists reserved (:434) before _materialize (:437) and before claimed is saved (:444). copytree / copy2 raising, or a stop in that window, leaves no receipt. The next serve allocates another attempt. When the new charge exceeds cap, the ticket is recorded rejected for budget exhaustion and never launches. When the cap has room, the same ticket runs under a second permanent charge while the first row stays reserved.
    • Trigger B: _save_ledger of finished (:464) happens before _record writes results/<ticket>.json (:465). A stop in that window has no claimed row, so the next serve launches build, candidate, and incumbent again.
      Expected correction: a persisted run_id ending in @ticket is that attempt. Resume reserved without a new charge. For claimed or finished, write a terminal receipt and do not launch.

Closed on this head (source inspection)

  • Submit refuses symlinks via a no-follow walk, and refuses an in-glob deletion instead of hashing a missing path.
  • serve_ticket drops the caller's now and re-reads clock or datetime.now(timezone.utc) for admission and each stage. Submit checks deadline_utc. A stage TimeoutExpired becomes an unsettled receipt.
  • claimed is saved before _launch. A later claimed row with no receipt is recorded unsettled and is not launched again. The reserved and finished windows above are what remains of that contract.
  • The namespace plan is built from the writable set, the readable set, runtime_roots(), and runtime_files() (PT_INTERP, ldd libraries, and /etc/ld.so.cache). Source / is refused. The coordinator environment is reduced to the allowlist. namespaces_available() probes unshare, and the live test is skipped when that probe fails.
  • now is optional on serve_ticket, so the CLI serve call matches the signature. A completed trial with an unreadable latency file becomes unsettled.

Follow-ups (non-blocking)

  • Each stage still uses min(seconds_remaining(...), 30) (parallel_round.py:357), so a stage stops at 30s while the lane deadline still has time left.
  • A symlink present at scan time is rejected. _scan and _read_nofollow do not fstat the opened fd and do not open with O_NONBLOCK, so a regular file replaced by a FIFO after stat can block the coordinator in os.read.

Verification

Static read of parallel_round.py, round_budget.py, confinement.py, device_lease.py, tools/parallel_round.py, and the new tests, checked against the earlier threads on #268. Not verified here: pytest, unshare, namespaces, models, devices, or a paid workflow. Loader mounts and OS confinement are not qualified by this pass. Synthetic latency in this harness is not a campaign result. M0–M2 scope acceptance stays with the maintainer.

Dong-bot will re-read after the author pushes.

@NumberWan NumberWan changed the title feat(harness): offline parallel-round coordinator feat(harness): 集群 RSI 并行轮协调器 Nov 11, 2026
@NumberWan

Copy link
Copy Markdown
Author

f63723f8 让一轮并行测量不再限定单设备、单指标:

  • 计划改为 device_ids 列表和 metric: {name, better}。serve 选任一空闲设备,把 {device} 代入 build / trial 命令,结果写下所用设备,gain 以正数表示候选更好。
  • 同一条 lane 的 ticket 先取得 lane 锁再查账本、取设备,多设备时不会并发改写同一份账本。
  • 结果文件里的指标不是有限数值时,留下 unsettled 收据,账本停在 claimed,不在 claim 之后抛异常。

pytest tests/test_round_budget.py tests/test_confinement.py tests/test_parallel_round.py:31 passed,1 skipped(本机没有 user namespace)。ruff check 通过。主 CLI 没有改动:按 docs/contracts/isolation-errors-and-cli.md,新的 CLI 入口等 #371 的 Apps 实现切片。

Copy link
Copy Markdown
Collaborator

增量 review:cf22cb50f243b17227863f8e79f601b25faf5480..f63723f81367991298913c3c626de135754f0754(1 个提交 feat(harness): let a parallel round use several devices and a declared metric),对照我上一条 review。这次还不 approve。

这次改动,已核对

  • prepare()(src/inferrsi/harness/parallel_round.py L167-184)校验 device_ids(非空、互不重复、单段名)和 metric: {name, better}。better 只接受 lower / higher,tests/test_parallel_round.py 新增的 test_prepare_rejects_bad_devices_and_metric 覆盖了这些拒绝分支。两个字段都写进计划正文,受 round_digest 约束。
  • serve_ticket()(L461-499)先拿 lane 锁,再在锁内重新检查 results/ 和账本里已有的 claimed,然后按顺序试各设备的 lease,都忙时返回 queued,不会预占 attempt。以前查 claimed 在拿设备 lease 之前、也不在 lane 锁里,多设备下同一 lane 可能并发改写账本,现在这个窗口关上了。test_a_busy_lane_stays_queued_without_touching_its_ledger 确认 lane 忙时不会生成 ledger 或 result。
  • _metric_value()(L362 起)用 type(value) in (int, float) 排除了 bool,非有限值会落成 unsettled,并且在 advance(..., "finished") 之前返回,账本停在 claimed。这与 F3 的约定一致,test_a_non_numeric_metric_is_an_unsettled_receipt 也覆盖了。better: higher 时 gain 取反,结果始终以正数表示候选更好,测试用 8 对 10 得 -2,符合预期。

这次新发现的问题

  1. _segment() 放过了 . 和 ..(L44-45)。device_ids: ["."] 或 lane id ".." 能通过 prepare()。到 serve 时,DeviceLease.acquire()(src/inferrsi/infra/device_lease.py L31-34)会执行 (directory / ".").open("a+"),抛出 IsADirectoryError。这不是 LeaseError,所以 serve_ticket 会直接抛异常,而不是返回 queued 或 rejected。账本不会被破坏(这时还没 admit),但这个计划永远跑不起来。请在 _segment() 里同时拒绝 . / ..,在 test_prepare_rejects_bad_devices_and_metric 里补这两个值,lane id 也补一个用例。
  2. {device} 只做了字符串替换,没有说明约定(_command,L355 起)。build 现在也会替换 {device},但不会替换 {result}。如果有人在 build 里写了 {result},会原样传下去。另外 metric.name 是结果文件里 metrics 的键,这个约定目前只出现在提交说明里。建议在模块 docstring 或 docs/ 对应的契约里写清楚 build / trial 可用的占位符,以及结果文件的 {"metrics": {<name>: number}} 形状。

上次的问题,还没处理
3. Ready CI 仍然没在这个 head 上跑:f63723f8 只有 declarations(2 个),没有 full-check (3.10/3.11)、installed-package 和 pr-references。31 passed / 1 skipped 仍是本地结果,请触发完整 CI。
4. CLI 测试断言仍然太宽:tests/test_parallel_round.py L541 还是 body["status"] in {"measured", "unsettled", "rejected", "queued"}。请按 namespaces_available() 分支断言具体状态,至少在不可用时断言 unsettled,并且 reason 指向 confinement 不可用。
5. Dong1017 的 CHANGES_REQUESTED 仍未关闭:这次没有改 src/inferrsi/infra/confinement.py,fresh root 的动态链接器闭合 thread(L99)和更早几条 thread 仍然没有作者回复,真实 user namespace 下 test_unshare_starts_python_and_hides_a_private_canary 也还没有人跑通过。请在对应 thread 里回复,并贴出能创建 user namespace 的环境里的运行结果。

以上是静态 review(diff 加 head 上的相关代码),我没有在本地跑测试。

@NumberWan

Copy link
Copy Markdown
Author

回复 @amy-why-3459 的增量 review,新 head 08a8de82:

  1. . / ..:_segment() 现在拒绝 ""、.、..。test_prepare_rejects_bad_devices_and_metric 补了 device_ids: ["."]、[".."],以及 lane id .、.. 两个用例。
  2. 占位符和结果文件约定:写进模块 docstring:build 只替换 {device};trial 替换 {device} 和 {result},候选与 incumbent 各跑一次,必须向 {result} 写入 {"metrics": {<metric.name>: <有限数值>}};其他文本(包括 build 里的 {result})原样传下去。
  3. Ready CI:f63723f8 上的 CI — ready / review lane 已经触发,状态是 action_required。这是 fork PR 需要维护者在 Actions 页面批准后才会运行,作者这边无法触发。请维护者批准 08a8de82 的运行。
  4. CLI 测试断言:test_serve_cli_runs_without_a_caller_supplied_clock 改为按 namespaces_available() 分支断言:可用时为 measured;不可用时为 unsettled,reason 等于 confinement backend unavailable。
  5. Dong1017 的 thread:已在各条 thread 里逐条回复对应的提交和测试。真实 user namespace 下的 test_unshare_starts_python_and_hides_a_private_canary 仍然没有结果:这台主机 max_user_namespaces=0,也没有可用的 docker daemon。需要一台能建 user namespace 的机器跑这条测试,在拿到结果之前,OS 隔离仍按未验证处理。

本地:pytest tests/test_round_budget.py tests/test_confinement.py tests/test_parallel_round.py 31 passed,1 skipped;ruff check 通过。

Copy link
Copy Markdown
Collaborator

增量 review:f63723f81367991298913c3c626de135754f0754..08a8de823e6a6fa342cd8bb5f926e28dd9e506f9(1 个提交 fix(harness): reject dot device and lane names, document trial placeholders,2 个文件),对照我上一条 review 和你的回复。这次还不 approve,原因见下面第 3、5 条。

已解决,已核对

  1. . / ..:_segment()(src/inferrsi/harness/parallel_round.py L49-54)现在拒绝 ""、.、..。它同时用于 device_ids(L180)和 lane id(L203-204),两条路径都覆盖到了。test_prepare_rejects_bad_devices_and_metric 补了 ["."]、[".."] 两个设备用例,以及 .、.. 两个 lane id 用例,lane 的断言 match="own id" 和 L204 的报错文字对得上。
  2. 占位符约定:模块 docstring(L6-9)写明 build 只替换 {device},trial 替换 {device} 和 {result},结果文件的形状是 {"metrics": {<metric.name>: <finite number>}}。和 _command()(L364-368)的实现一致:result is None 时不替换 {result}。
  3. CLI 断言:test_serve_cli_runs_without_a_caller_supplied_clock 现在按 namespaces_available() 分支断言,不可用时断言 unsettled,并断言 reason 是 confinement backend unavailable,和 src/inferrsi/infra/confinement.py L265 抛出的文字一致。

仍未关闭
3. Ready CI 还没在这个 head 上跑:08a8de82 上只有 declarations 和 knowledge declaration policy 通过;CI — ready / review lane 状态是 action_required,需要维护者在 Actions 页面批准。full-check (3.10/3.11)、installed-package 和 pr-references 跑完之前,测试结果只能算作者本地的(31 passed / 1 skipped)。
5. 真实 user namespace 下的隔离仍未验证:这次你已在 Dong1017 的各条 thread 里逐条回复了(包括 confinement.py L99 的加载器 thread),这部分我认可。但 test_unshare_starts_python_and_hides_a_private_canary 仍然没有在能建 user namespace 的机器上跑通,Dong1017 的 CHANGES_REQUESTED 也还在。另外要注意,在 CI 和你的主机上,CLI 测试的 measured 分支和真实的 UnshareBackend 路径目前都不会被执行。需要有人在能创建 user namespace 的环境里跑一次这条测试和 test_serve_cli_runs_without_a_caller_supplied_clock,并贴出结果,再由 Dong1017 确认 thread 是否可以关闭。

以上是静态 review(diff 加 head 上的相关代码),我没有在本地跑测试。这次是校验和文档改动,不涉及精度和性能对比。

Reviewed: f63723f81367991298913c3c626de135754f0754 → 08a8de823e6a6fa342cd8bb5f926e28dd9e506f9.

When a timed-out command's process group does not empty within the
reap window, confinement raises UnreapedCommand. serve now writes
quarantine/<device> for every device that ticket held, and
hold_devices skips quarantined devices until the file is removed.
Releasing the flock alone was not enough: the next ticket could be
handed a device the old payload still used, and a one-shot serve
process drops its flock on exit anyway.

The _kill_group docstring now says what it checks: the process group
is empty. Processes that called setsid are covered by --kill-child
tearing down the PID namespace.

Signed-off-by: NumberWan <wantszkin2003@gmail.com>
The confined command is PID 1 of its namespace. If it calls setsid it
leaves unshare's process group, so killing the group took down unshare
only; _kill_group then saw an empty group and returned while PID 1 could
still hold the device node, and serve released the lease.

On timeout, _kill_group now records unshare's direct children (PID 1)
with their start times, kills the group and each child, and returns only
once the group is empty and every recorded child is gone or a zombie. A
zombie PID 1 has already outlived every other process in its namespace.
If that does not happen within the reap window, UnreapedCommand is
raised and serve quarantines the devices.

Signed-off-by: NumberWan <wantszkin2003@gmail.com>
Quarantine now lives next to the lease file as
leases/<device>.quarantined, written by DeviceLease.quarantine while the
lease is still held. DeviceLease.acquire checks the marker after taking
the flock and refuses the device until the file is removed, so every
acquire through the lease store sees it, not only hold_devices.

Signed-off-by: NumberWan <wantszkin2003@gmail.com>
_kill_group signals a recorded namespace PID 1 only if /proc still shows
the same start time and it is not a zombie, so a recycled pid is left
alone. _proc_stat returns None for a short /proc read instead of
raising before the kill.

Signed-off-by: NumberWan <wantszkin2003@gmail.com>
serve no longer has its own ABBA loop. After the confined build, a ticket
is measured by run_experiment with a confined execute, and evaluate gives
the verdict, as issue InferMatrixLab#249 asks (reuse run_experiment, evaluate and
qualify_devices; no second runner).

- The trial adapter keeps the lane deadline, command timeout, device view
  and read-only repository; an unreaped command still quarantines the
  held devices.
- The receipt carries verdict, improvement_pct, reasons, medians, samples
  and run_dir; a ticket without a complete metric set stays unsettled.
- Optional plan fields: validation (run once on the candidate),
  metric.min_improvement_pct (default 0), and device_model, which runs
  qualify_devices on the held devices before an attempt is reserved and
  leaves the ticket queued when the check fails.

Signed-off-by: NumberWan <wantszkin2003@gmail.com>
Add close that overlays PASS-lane outboxes in lane_id order, measures the
joint candidate once with run_experiment, and writes successor/ only on
PASS. This is an intentional MVP, not InferMatrixLab#249 stepwise composition.

Also stop host git on candidates after a writable build (_tree_identity)
and map ConfinementUnavailable during validation to unsettled instead of
measured FAIL, matching the review blockers on 026f686.

Signed-off-by: NumberWan <wantszkin2003@gmail.com>
After rebasing onto main, RunSpec lives in inferrsi.contracts and
run_experiment requires assess/report ports. Wire the v0 gates from
benchmark.legacy_run_v0 and keep report as a no-op; measurement still
reads evaluation.json.

Signed-off-by: NumberWan <wantszkin2003@gmail.com>
@NumberWan

Copy link
Copy Markdown
Author

be9af6f5:已将 parallel-round-offline rebase 到当前 main,并适配新套件结构。

  • RunSpec → inferrsi.contracts
  • 测量仍走 harness run_experiment 循环;assess 用 benchmark.legacy_run_v0 的 v0 门禁,report 为空操作(仍读 evaluation.json)
  • 本机:tests/test_parallel_round.py 38 passed;editable install 下 collect 不再出现 No module named 'inferrsi.runspec'

请再看一次 CI full-check。GPU 实机证据仍独立于这次 rebase。

@NumberWan

Copy link
Copy Markdown
Author

be9af6f 已 rebase 到当前 main 并适配 contracts/新 run_experiment ports。请 @amy-why-3459 @Dong1017 在新 head 上再看一次;先前 head 上的 review 已因 push 变为 stale。

cursor[bot]
cursor Bot previously approved these changes Nov 17, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 1acfacf8d49a13edf8d1fbdfde224c11ddbee8bf and head be9af6f579ec58c989d20d2a3d8ee08a85d98e85.

Independent re-review after rebase dismissed the prior Approve on 2da25057. Noon P1 (_tree_identity / planted-git) and P2 (validation ConfinementUnavailable → unsettled) still hold with regressions. Tip be9af6f adapts RunSpec/assess/report ports (fixes prior Ready import break). Local focused tests 58 passed / 1 skipped; ruff clean; declarations green. Ready full-check on this tip still needs to finish after fork approval — merge held for that plus a second independent LGTM. Does not close #249 / M0–M2.

@Dong1017

Copy link
Copy Markdown
Collaborator

Posted by an authorized steward agent on behalf of @Dong1017 (hosted Dong-bot on usage_limit).

Steward re-review (afternoon): head moved off noon 2da25057 → be9af6f579ec (rebase + port adaptation). Independent cursor[bot] Approve on current head: #268 (review)

Noon P1/P2 still hold. Tip adapts RunSpec/assess/report (addresses prior Ready import break). Fork Ready run approved and requeued: https://github.com/InferMatrixLab/InferRSI/actions/runs/44028917826

Merge held for (1) Ready full-check green on this tip and (2) a second qualifying independent LGTM. Does not close #249 / M0–M2.

@NumberWan NumberWan changed the title feat(harness): 集群 RSI 并行轮协调器 feat(harness): 離線集群 RSI 并行轮协调器 Nov 17, 2026

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR review verdict — DO NOT PASS

Independent blind review of base 1acfacf and head be9af6f. Submitted as COMMENT, not REQUEST_CHANGES, under the owner's review-action constraint. No other reviewers' comments or reviews were read.

Blocking findings:

  1. [P1] New files bypass the frozen protected/disjoint lane boundary (src/inferrsi/harness/parallel_round.py:313–332, 388–390). prepare checks protected intersections and differently spelled overlapping globs only against files already in the incumbent; submit checks only the submitting lane's globs. I reproduced a plan with lanes new/* and new/*.txt, protected new/protected.txt, and an incumbent with no new/ directory. prepare accepted it, then BOTH lanes successfully submitted new/protected.txt. Thus adding a file can modify a protected path and give two lanes ownership of the same path. Reject protected paths and ambiguous cross-lane ownership on each concrete submission, including newly created files; add regression coverage.

  2. [P1] The namespace runtime omits the base interpreter's standard library when launched from a virtualenv (src/inferrsi/infra/confinement.py:115–123). runtime_roots includes sys.prefix (the venv) and the resolved executable's bin directory, not sys.base_prefix / the standard-library tree. On this Linux host namespaces_available() is True, but real UnshareBackend execution from a Python 3.12 venv fails at interpreter startup with ModuleNotFoundError: No module named 'encodings'. The CLI returns unsettled instead of measured. Include the necessary base runtime/stdlib locations without exposing unrelated host paths, and test actual namespace execution from a venv.

Executed:

  • Read the eight-file diff, new coordinator/budget/confinement/lease implementation, CLI, affected tests and runner integration; followed AGENTS.md and review/contribution guidance, with the owner's blind-review and COMMENT restrictions taking precedence.
  • Focused pytest: tests/test_parallel_round.py tests/test_round_budget.py tests/test_confinement.py: 53 passed, 5 failed, 1 skipped. Failures: serve CLI; live unshare startup/canary; both timeout-child-cleanup cases; device-node/proc/scratch test. These failures occur in the venv namespace-runtime path above; they are not claimed as five independent defects. NVIDIA/torch test skipped.
  • Changed-file ruff check: passed. git diff --check: passed.
  • Independent temporary-directory scope reproduction above: both forbidden submissions accepted.
  • Exact-head check-run metadata includes two failures; I did not inspect their logs or infer their causes.

Coverage/acceptance limits:
This is a substantial 3,679-line, eight-file addition. Full repository pytest, accelerator execution, performance/pilot evidence, and every concurrency/security case were not validated. Original #249 requirements were supplied through an isolated requirements-only extraction, not mixed review discussion. This PR explicitly implements a narrower offline single-round slice, without the drivers/pilot and with one joint rerun rather than stepwise select-compose-rerun. Acceptance of that scope reduction remains unresolved; this review does not clear #249 or claim campaign PASS. Fix the reproduced defects, establish the accepted slice, and rerun the relevant live boundary tests before approval.

Move parallel-round assess composition into apps so harness stays clear of
benchmark imports. Reject protected and cross-lane paths on concrete
submit, and mount the versioned stdlib for UnshareBackend under a venv.

Signed-off-by: NumberWan <wantszkin2003@gmail.com>
@NumberWan

Copy link
Copy Markdown
Author

@zengchuang-hw 已在 ceb0f1f1 处理两条 P1:submit 对具体路径(含新建文件)拒绝 protected 与跨 lane overlap,并补回归;runtime_roots 挂入 versioned stdlib(不挂 /、/usr),并增加 venv 下可 import encodings 的断言/实跑测试。同提交还将 parallel-round 的 assess 组装移到 apps/,消除 harness→benchmark 的 import-gate 违例。请在新 head 上再看一次。

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR review verdict — DO NOT PASS

Independent blind incremental review of be9af6f → ceb0f1f; current target base 1acfacf. COMMENT is intentional under the owner's action constraint. No other review/comment bodies or author retellings were consulted.

Prior findings rechecked:

  • The concrete protected/new-file and cross-lane ownership bypass is fixed in submit. The added regression exercises refusal for both lanes.
  • The previously failing Python 3.12 virtualenv namespace path is fixed on this host. Real namespace startup, canary isolation, CLI serve and timeout cleanup tests now pass.
  • Inspected the new apps assess adapter and its delegation; run_rules carries the same RunSpec fields as the previous inline conversion. Read all eight changed files in this incremental diff, including the architecture-test expectation changes.

Remaining reproduced defect:
[P2] The new no-/usr runtime test fails with a normal system interpreter (src/inferrsi/infra/confinement.py:124–143; tests/test_confinement.py:239–244). Although the new base-prefix branch excludes /usr, candidates still starts with sys.prefix. For /usr/bin/python3, sys.prefix == sys.base_prefix == /usr, so runtime_roots() returns (/usr,) and the new assertion fails. I ran this exact new pytest test with the system Python 3.13 interpreter and observed the failure. Apply the broad-root exclusion to every candidate, not just the added base prefix, while retaining the interpreter/stdlib paths needed for system Python; cover both system and venv execution. This also aligns the actual mount set with the new function's stated preference not to expose a broad /usr tree.

Validation:

  • Python 3.12 venv: test_parallel_round.py, test_round_budget.py, test_confinement.py, test_knowledge_compatibility_shells.py, test_target_module_import_gate.py: 90 passed, 1 skipped, 43 subtests passed. Only NVIDIA/torch hardware case skipped.
  • System Python 3.13: test_runtime_roots_include_stdlib_without_mounting_usr failed with '/usr' in roots. Reused available pure-Python pytest dependencies via PYTHONPATH; repository source was this exact head.
  • Incremental changed-file ruff and git diff --check: passed.
  • Exact-head check-runs lookup returned one successful check. This is not evidence that all required lanes/checks completed; logs and required-check configuration were not examined.
  • Full repository suite, real accelerator/pilot and exhaustive concurrency/security validation not executed.

Acceptance remains limited to the reviewed offline slice. The earlier requirements-only extraction of #249 required stepwise composition; this code still performs a single joint rerun, and no verified accepted scope reduction has been provided. Drivers/pilot and overall #249 acceptance remain outside the demonstrated evidence. The two prior reproduced P1s are cleared for their tested conditions, but the newly failing supported-interpreter test and unresolved scope evidence prevent approval.

System interpreters use prefix=/usr; filter that broad root for all
candidates and keep versioned stdlib plus the executable directory.
Cover both the active interpreter and a /usr/bin/python3 subprocess.

Signed-off-by: NumberWan <wantszkin2003@gmail.com>
@NumberWan

Copy link
Copy Markdown
Author

@zengchuang-hw 已在 07817ac6 处理后续 P2:runtime_roots 对每个 candidate 都排除 / 与 /usr;系统解释器保留 versioned stdlib 与 executable 目录。测试覆盖当前解释器,并以 /usr/bin/python3 子进程覆盖系统 Python。请在新 head 上再看一次。

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR review verdict — DO NOT PASS (acceptance evidence incomplete)

Independent blind incremental review of ceb0f1f → 07817ac, against target base 1acfacf. COMMENT is intentional under the owner's action constraint.

The previous /usr finding is resolved: the broad-root exclusion now applies to every candidate. I inspected the actual two-file executable-code delta through an AST view excluding comments/docstrings, and tested both interpreter configurations. No new defect was found in this incremental fix. This clears the prior P2 for the tested system and virtualenv configurations; it is not a claim of complete PR acceptance.

Executed on this exact head:

  • Python 3.12 virtualenv: tests/test_parallel_round.py, test_round_budget.py, test_confinement.py, test_knowledge_compatibility_shells.py, test_target_module_import_gate.py: 91 passed, 1 skipped, 43 subtests passed.
  • System /usr/bin/python3 (3.13), with available pytest dependencies supplied through PYTHONPATH: entire test_confinement.py: 20 passed, 1 skipped. This includes the formerly failing /usr assertion and actual namespace execution/startup, canary and timeout tests.
  • Both skips require NVIDIA hardware and torch; no accelerator execution was performed.
  • Changed-file ruff and incremental git diff --check: passed.
  • AGENTS.md and the previously read review skill are unchanged in this increment.
  • Exact-head check-runs lookup returned one successful check. I did not verify all required-check configuration, full repository tests, accelerator/pilot evidence or exhaustive security/concurrency behavior.
  • Before submission: PR open, author NumberWan, head unchanged; no existing review by this account on this exact head.

Remaining acceptance gap, independently checked:
The isolated original requirements for #249 specify stepwise composition with a fresh remeasurement after each passing lane addition. Current close_round still materializes all selected overlays together (line 1067), calls _run_experiment once (line 1093), and then writes the successor (line 1140). The incremental fix does not change that behavior. No verified accepted reduction of that original requirement to one joint rerun is available to this review. The broader drivers/pilot deliverables are also not established by these tests. Therefore the code findings from my earlier reviews are cleared in their tested scope, but approval of this whole PR or closure of #249 would still overstate the acceptance evidence. Establish the accepted delivery slice or satisfy the original composition requirement before requesting overall approval.

No other reviewers' opinions, author comment bodies or author retellings were read.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 1acfacf8d49a13edf8d1fbdfde224c11ddbee8bf and head 07817ac6f2c424687021f676dd3a18bbec425929 (offline prepare/submit/serve/close slice for #249; not M0–M2 closure).

Verified on tip vs afternoon be9af6f:

  • Submit now refuses protected and cross-lane-owned concrete paths (including newly created files); regression test_submit_rejects_new_protected_and_cross_lane_owned_paths present.
  • runtime_roots excludes broad / and /usr for every candidate while keeping versioned stdlib; venv + system-interpreter coverage added.
  • Assess composition moved to apps/parallel_round.make_assess; Harness takes an injected make_assess port (no Benchmark import in harness).
  • Prior noon P1/P2 (planted .git / validation confinement → unsettled) still hold from earlier tips.
  • Local: test_parallel_round + test_confinement 59 passed / 1 skipped; ruff clean on changed paths; real-tree gate reports 21 harness→model/infra hits including parallel_round as this tip asserts.
  • Ready lane on tip was action_required; steward approved workflow run 44045958946 (queued). Declarations success on tip.

Scope note (not a code defect): joint single-rerun close is the accepted delivery for this PR; stepwise per-lane remeasure and full #249 M0–M2 remain open. Merge held for a second current independent LGTM + Ready green on tip; gate assertions will need rebase vs R2 (#478) / current main.

@Dong1017
Dong1017 dismissed stale reviews from themself November 17, 2026 09:41

Stale: findings from Oct/early-Nov heads superseded by tip 07817ac (submit protected/overlap, confinement runtime_roots, assess port). Steward re-approved current head.

@Dong1017

Copy link
Copy Markdown
Collaborator

Posted by steward agent for @Dong1017 (hosted Dong-bot on usage_limit).

Re-review at head 07817ac6: PASS → Approve

@Dong1017 decide — accepted delivery slice for this PR

Accept joint single-rerun close (compose selected PASS overlays once, one remeasure, keep/retain) as the bounded offline slice for PR #268. This is not #249 M0–M2 acceptance and does not authorize stepwise per-lane remeasure or milestone closure. Stepwise composition remains open on #249.

Merge held for a second current independent LGTM and Ready full-check green on this tip. Expect a rebase against current main / R2 (#478) for gate hit-count assertions before merge.

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR review verdict — DO NOT PASS (remaining acceptance gap)

Explicit response to the two review requests:

Current reviewed head: 07817ac; base: 1acfacf. This explicitly answers both requests rather than assuming a later review or unchanged head proves request coverage. I inspected only request metadata, my own review records and clean source/requirements, not requester opinions, author retellings or other reviews.

This pass rechecked the executable AST of submit, runtime_roots and close_round against the original isolated #249 requirements. There is no source delta since my preceding exact-head review. Concrete protected/cross-lane submission checks and the all-candidate /usr exclusion remain present. The previous findings recorded in 5934315001 and 5934770356 remain cleared for the already tested conditions.

The remaining gap is unchanged and was rechecked in code, not inferred from discussion: close_round materializes all passing overlays together at line 1067, performs one joint _run_experiment at line 1093, then writes successor at line 1140. That does not implement the original stepwise fresh-checkout remeasurement after each lane addition. No verified acceptance of the narrower delivery slice is available. Consequently I cannot approve the complete PR or clear #249 from this evidence.

Fresh checks in this pass: relevant ruff and git diff --check passed; exact-head check-runs lookup returned ten distinct checks, all successful. I did not inspect the required-check configuration or CI logs, so this is not a declaration that every required acceptance condition is complete.

Previously executed on this exact head, not rerun in this pass: five relevant test groups under Python 3.12 venv: 91 passed, 1 skipped, 43 subtests passed; entire confinement group under system Python 3.13: 20 passed, 1 skipped. Both skips require NVIDIA/torch. Full repository tests, accelerator/pilot evidence and exhaustive concurrency/security verification remain unperformed. This follow-up ran no device experiment and changed no repository code.

Both requests are now answered with this revision-bound COMMENT. That closes the response gap, not the technical/acceptance gap; COMMENT rather than REQUEST_CHANGES follows the owner's action constraint.

@Dong1017

Copy link
Copy Markdown
Collaborator

Steward note (authorized delegate for @Dong1017): tip 07817ac6 cursor LGTM and Ready green still stand for the offline #249 slice, but after #478 (R2) landed on main this PR is now DIRTY/CONFLICTING (behind_by includes the R2 gate-count change; tip still asserts harness reaches model/infra 21 hits). Dong second Approve + merge are deferred until rebase onto current main and a tip re-check of the import-gate expectations. Do not force-merge over conflicts.

@Dong1017

Copy link
Copy Markdown
Collaborator

Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

The steward note is correct, and I'm not approving 07817ac6. This PR is DIRTY/CONFLICTING against main after #478 (R2), so my second Approve and the merge stay deferred. Don't force-merge over the conflicts.

What blocks merge now

  1. Rebase onto current main. behind_by includes the R2 gate-count change, and the tip still asserts harness reaches model/infra = 21 hits. Once that count changes, the expectation fails or the conflict resolution silently picks the wrong number. @NumberWan, please rebase and resolve against the R2 number. Don't just retype 21.
  2. Re-check the import gates on the new tip. Confirm the apps-side assess assembly from ceb0f1f1 still keeps harness→benchmark out of the gate, and that the hit-count assertion matches what main now reports.
  3. Fresh Ready/full-check on the rebased head. The earlier green run (44045958946) was for 07817ac6 and does not carry over to the rebased head.
  4. A second independent current-head LGTM. A new push dismisses earlier Approves, so this has to be on the rebased head.

Review state

I did not re-review any code. There is no new diff to inspect yet. Re-ping @Dong1017 once the rebased head is pushed. I'll do an incremental re-check and submit a formal verdict then.

(Dong-bot, authorized delegate for @Dong1017.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:harness Harness module: profile, A/B loop, quality gates, evidence reports bot-reviewed kind:task Bounded delivery task with a delegated slice

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants