Skip to content

task: B01 — 旧 run/evaluate 整链迁入最终归属 #414

Closed

Description

@Dong1017

已确认实施方案:用户于 2026-11-12 11:35–11:36(北京时间)确认综合推荐,并要求快速、干净切换。

  • 目的:首次创建 benchmark/,把旧 run 链按职责拆完:规则进 B,执行事实留 H,组合进 Apps legacy。并删除该链上的 H→model 以及除进程域之外的 H→infra 边。
  • 迁入或删除:harness/evaluation.py 的 evaluate / _summary 进 B;runner 不再内嵌判分与写报告,改由 Apps legacy 组合;scenarios/external 的执行留 H、metric 校验进 B;reporting.write_report 进 Apps;collect_samples 进 B 采样(D008 未列此函数,本 PR 须先补 owner 再搬)。留在 H 的写入改用 H 自有原子 JSON(与 infra.artifacts 字节等价)。model/protocols/metrics 可先由 B 导入,物理迁移留给后续 validators 切片。旧 harness 路径不留会形成 H→B/Apps 的壳;根 evaluation.py / execution.py / reporting.py 可暂留精确转发,到 R01 再删。
  • 前置:H02、Publish standardized Apps run output #397;task: Apps 应用层详细设计与迁移计划(CLI / Python / Plugin) #371 接受 apps/legacy/ 路径。
  • 验收:删除 evaluation/external 的 model 边,以及 runner/external/media 对 artifacts/targets 的边;旧 run 退出码 0/1/2、incomplete 优先、同一夹具下 evaluation.json 字节不变;absent: benchmark/ 消失;overall 仍为 false。
  • 明确不做:skill-eval 族、B 的 prepare/assess 生产实现、model/validators 迁移、根转发退役、CLI 迁移。

合入后 overall isolation_complete 必须仍为 false。

关联架构总题 #387、Apps 设计 #371;#389 是当前清单与 observability 来源,#398 为已合入 H01,#397 保持纯 Apps 标准输出且只读不扩范围。

Activity

  1. added
    kind:taskBounded delivery task with a delegated slice
    area:harnessHarness module: profile, A/B loop, quality gates, evidence reports
    on Nov 12, 2026
  2. amy-why-3459 commented on Nov 12, 2026

    @amy-why-3459
    Collaborator

    谢谢,B01 的拆分思路清楚。我对照了当前 main(e2f4cd1e)上的相关文件,补充几点:

    1. harness/runner.py 现有的边(L11-16):infra.artifacts.atomic_write_json、infra.targets.repository_identity、.evaluation.evaluate、.reporting.write_report、.scenarios.external。正文覆盖了 evaluate、reporting 和 artifacts。repository_identity 归谁(H 自备,还是由 Apps legacy 组合时传入)建议在 PR 里写明,因为验收要求"runner 对 artifacts/targets 的边"都消失。
    2. harness/evaluation.py 还导入了 ..claim.Verdict(L5)和 ..runspec.RunSpec(L7)。evaluate / _summary 搬进 B 后,B→claim/runspec 这两条边是否合规,请对照 rfc: K/H/B 与 Apps 公开接口详细设计(最小交接与模块边界) #387 的目标图确认,必要时一并登记。
    3. "evaluation.json 字节不变"和"退出码 0/1/2、incomplete 优先",建议各配一个迁移前后对比的夹具测试,至少覆盖 pass、fail 和 incomplete 三种情况。
    4. 前置条件:task: Apps 应用层详细设计与迁移计划(CLI / Python / Plugin) #371 仍是 status:proposed,Publish standardized Apps run output #397 仍是 Draft,main 上还没有 apps/。正文已把"task: Apps 应用层详细设计与迁移计划(CLI / Python / Plugin) #371 接受 apps/legacy/ 路径"列为前置,这一点很好,请在开工前确认。
  3. Dong1017 commented on Nov 15, 2026

    @Dong1017
    CollaboratorAuthor

    @amy-why-3459 @zengchuang-hw — under the delegated architecture-coordination role, I adopt the joint five-entry D008 record for these two governing issues:

    1. Benchmark owns sampling order, timing and artifact hashes; transport stays caller-injected.
    2. Harness owns the required snapshot port; Apps binds the infra implementation, with no Harness concrete default.
    3. Apps owns workflow display projections.
    4. Harness owns factual profile records, bounded single-trace overview and integrity/digest reads; Apps owns UI and live-route decoration.
    5. Apps owns pre-profile search display projections.

    B01 delivers its implementation plus this shared decision metadata; OBS retains its implementation. This records architecture adoption, not independent implementation acceptance. Both issues retain their evidence requirements; isolation_complete remains false.

  4. Dong1017 commented on Nov 15, 2026

    @Dong1017
    CollaboratorAuthor

    Closing the bounded B01 migration slice delivered by PR #422, merged as 9ede1835.

    The legacy run/evaluate chain is assigned to Benchmark, Harness facts and Apps composition, with compatibility retained. Evidence includes hosted checks and the PR's exact-head Linux regression. The shared D008 record was adopted on #414 and #413.

    A-entry, R01 and #410's formal main-only replay remain separate. Overall isolation_complete remains false.

  5. zengchuang-hw commented on Nov 17, 2026

    @zengchuang-hw
    Contributor

    Independent read-only follow-through for comments:6887366262:2026-11-15T01:34:04Z. Issue414 remains closed. I found no existing own comment explicitly covering this request. I used the isolated original-goal extraction and current executable code, not the source's adoption/delegation assertions or other reviews.

    Baseline inspected: main 0311393c62330c041a01238dc86823b80e020561. The safe scope is the legacy-run responsibility split and preservation of behavior, not wholesale core isolation or reopening implementation work.

    Current evidence: apps/legacy/evaluation.py converts a RunSpec into Benchmark rules and delegates evaluation to benchmark/legacy_run_v0.py. The latter owns the verdict calculation and explicitly gives incomplete evidence precedence over failure and success. Current harness/atomic_json.py owns a local JSON writer with Unicode, indentation, finite-number enforcement and trailing newline. The selected tests exercise the writer's byte equivalence, injected execution/assessment ports, metric-validation ownership, retired entry behavior, legacy exit codes, and frozen evaluation fixtures. This is evidence for these concrete boundaries; it is not an assertion that a historical adoption document is authoritative.

    Fresh actual checks: on this exact main revision,

    PYTHONPATH=src python -m pytest -q -o addopts='' \
      tests/test_legacy_run_split.py \
      tests/test_legacy_run_v0_frozen.py \
      tests/test_target_module_import_gate.py
    

    completed 50 passed, 43 subtests passed, one pytest-plugin deprecation warning, in 21.13s. These are bounded CPU/synthetic checks, including frozen evaluation-byte/exit-code cases, not model/device executions.

    A separate current-tree gate read returned zero violations but isolation_complete=false, with blockers for 12 Harness→Infra hits and five runtime-resource loaders. This distinction matters: the legacy split tests passing does not establish global isolation or authorize removing those blockers. Current retirement of old entrypoints is also a later repository fact, not grounds to retroactively expand this Issue's original scope.

    Remaining limits: the exact pre-adoption requirement version and authority for the referenced ownership decision were not independently recovered. I did not infer acceptance of prerequisite Issues or the excluded skill-eval, production prepare/assess, model/validator migration, CLI migration or root-retirement work. Full-suite, installed-package, native Windows and historical source-to-candidate equivalence outside the selected frozen fixtures were not run here. No comprehensive prerequisite closure or whole-Issue PASS is asserted.

    AGENTS/review guidance was checked. No code, tracker, status, policy or artifact was changed; no closed task was reopened, no merge occurred, and no model/device/paid experiment or previously stopped task was resumed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:harnessHarness module: profile, A/B loop, quality gates, evidence reportskind:taskBounded delivery task with a delegated slice

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions