Skip to content

refactor(apps): group application workflows in a workflows package - #504

Open
zengchuang-hw wants to merge 1 commit into
InferMatrixLab:mainfrom
zengchuang-hw:codex/apps-490-workflows
Open

zengchuang-hw wants to merge 1 commit into
InferMatrixLab:mainfrom
zengchuang-hw:codex/apps-490-workflows

Conversation

@zengchuang-hw

@zengchuang-hw zengchuang-hw commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Refs #490
Refs #371

Governing Issue and delivery slice

Apps 根目录的五个工作流实现归入 apps/workflows/,CLI、展示和评测工具同步使用实现所在路径。旧 Python 导入及工程校验脚本入口保留。

这是 #490 已接受方案的第三步,共五步。前两步 storage(#493)和 handoff(#496)已合入;本 PR 基于 main 965ab5cf83399a99530949b04bb934a892ee417c,head 为 940d54ee365769bedefc117530c74ba3eb88345f。接受记录见 Dong1017 的设计确认,实施范围及运行时摘要路径补充见 开工记录。

Changed artifacts and evidence

  • 迁移 campaign、benchmark_operations、improvement、skill_evaluation、skill_eval_engineering 五个实现。与 base 原实现逐一比对,规范化换行后的完整源码完全相同。初始化文件不导入子模块;仓库调用方和私有补丁切换到 canonical 路径。
  • 五个旧模块显式静态导出原有公开函数,使用字面量 __all__。迁移清单累计 12 项,校验对象身份、旧 pickle 加载、新定义位置、消费者登记、storage 依赖方向及导入副作用。
  • 工程校验器原来支持 python -m 和按文件执行;其 shim 使用静态绝对导入及 __main__ guard,清单用 script_entry: main 登记这一入口。仓库外 wheel 检查覆盖新旧两个模块、两种启动方式,比较退出码、空标准流及 result.json 字节。
  • Harness 的修改仅为 _runtime_identity() 中 campaign 源文件路径,改为 apps/workflows/campaign.py;评测工具的 controller 快照也指向实际实现。完整摘要表校验继续拒绝不匹配的 runtime。
  • 按 David 的跨版本恢复意见,补运行时边界回归和操作说明:两个独立 wheel 环境中,旧版创建并中断运行,新版的 status/recover 均拒绝且所有留存文件字节不变;旧版恢复同一 attempt,候选执行计数为 1;新版新计划包含 canonical 源码摘要。未取得固定旧 wheel 时,该安装集成用例明确跳过;普通回归仍验证旧摘要路径拒绝、计划字节保留及 canonical 源码摘要。

Checks and dependencies

Windows / Python 3.12 本地验证,以下为独立运行结果,存在覆盖重叠:

检查 结果
Ruff(src/tests/tools)、diff whitespace、五个实现源码等价比对 通过
迁移、session/project 优化、工程校验及 comparison 核心回归 212 passed,6 skipped(包括未配置旧 wheel、Linux/sandbox 条件用例)
新增入口及摘要/迁移补充检查 31 passed;最终清单检查另跑 24 passed
workflow 调用方、改进/评测、检索和导入边界扩展回归 546 passed,330 subtests passed;6 failed,2 errors,均已在未改动 base 复现,详见下方
optimization story、smoke、claim 49 passed;1 skipped(本机未安装 Playwright)
Codex / WorkSwarm 独立包、原生 WorkSwarm launcher、DSH 打包及桥接 55 passed;1 条现有 tarfile 弃用警告
独立 sdist → wheel → 新环境、仓库外 -I check_installed.py 通过;930 个资源文件与源码清单一致,128 条知识记录可读
固定迁移前 wheel 与新 wheel 的恢复边界集成 1 passed,实际执行;旧 wheel 源码取自上述 base,新 wheel 取自本变更的 runtime 源码

扩展回归的 Windows 基线缺口未改动:

  • CRLF 元数据用例 1 项失败:base 同样失败;仅在测试进程内关闭 Git 自动换行转换后,两种换行用例均通过,没有修改用户配置。
  • 手工 artifact 合成控制 5 项失败:Windows write_text 转换换行,合成判定中的 LF 字符串不匹配;同样五项在 base 复现。
  • 超大 HTTP body 参数用例发生 setup/teardown 两个错误:pytest 的测试名超过 Windows 环境变量长度上限;base 同样复现。超长合成参数不复制进 PR 文本。

构建依赖的联网下载遇到间歇 TLS 失败,随后使用已缓存的公开依赖完成隔离构建与安装验证。没有修改依赖、CI 车道、冻结证据或历史 validator。当前变更不触及 Knowledge authorship policy 的受保护路径。

Remaining work and Issue closure

本 PR 交付 workflows 切片。#490 第四步 integrations 和第五步集成验收仍待实施,#371 仍跟踪整体 Apps 落地及生产联调。Issue 完成由 Apps 负责人结合接口 owner 的检视确认;本 PR 不宣告整个 Issue 完成。

请 Dong1017 复核 Harness 摘要路径与恢复边界,david6666666 复核跨版本意见的落实;Amy 可补充独立检视。Ready 状态提交,合入依现行两份有效独立审批及检查结果执行 squash。

Signed-off-by: zengchuang <zengchuang3@huawei.com>
Agent-assisted-by: Codex (author)

Copy link
Copy Markdown
Contributor Author

@Dong1017 @david6666666 @amy-why-3459 请独立检视 #490 第三步 workflows,当前 head 940d54ee365769bedefc117530c74ba3eb88345f,Ready 提交。

Dong 请重点复核 Harness 摘要文件切换、CLI 按需导入及原运行恢复约束;David 在 #490 提出的跨版本意见已补真实双 wheel 回归:新版本拒绝旧计划且所有留存文件不变,固定旧版本恢复原运行,候选只执行一次。Amy 请补充独立检视新旧导出、脚本入口和插件安装兼容性。

五个实现的规范化源码与 base 完全相同;已完成核心回归、55 项三插件检查、仓库外安装及双版本恢复。扩展验证中的 Windows 基线失败已逐项在未改动 base 复现,并如实记录在 PR 描述。若有问题会先处理;通过后请对当前内容提交正式审批,按仓库规则两份有效独立审批且检查通过后 squash 合入。

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 965ab5cf83399a99530949b04bb934a892ee417c and head 940d54ee365769bedefc117530c74ba3eb88345f (full diff, first review). There is no earlier formal review and no outstanding CHANGES_REQUESTED, so this Approve is the first formal review on this head.

Verified from the diff

  • Moves are byte-identical. The five new modules under apps/workflows/ (campaign, benchmark_operations, improvement, skill_evaluation, skill_eval_engineering) are the old files moved unchanged. The new-file blob hashes match the old index hashes, e.g. b30c2aef, 7770c3f5, 187329e7, 06482510 and b907e9a5. No behavior or signature change is hidden in the move.
  • Old paths are shims that match the inventory. Each old module now holds only a docstring, an explicit import and a literal __all__. The __all__ order matches the exports of the new entries in docs/apps-module-migrations.json.
  • The engineering verifier shim keeps its script entry. It imports the canonical module by absolute name and keeps the if __name__ == "__main__": main() guard. The inventory records this as script_entry. test_shim_is_an_explicit_export_of_the_same_objects handles the four-statement shape and the level-0 import.
  • Script entries are tested. tests/test_skill_eval_engineering.py runs both module paths via -m and via file path. tests/check_installed.py repeats this against the installed wheel and asserts identical result bytes.
  • Internal imports are updated. workflows/skill_evaluation.py still uses from . import skill_eval_engineering, which now resolves inside workflows. The CLI, capture_delivery and optimization_story views, and the tools all import the canonical paths.
  • Frozen-runtime fingerprints are consistent.
    • harness/optimization_session._runtime_identity now fingerprints apps/workflows/campaign.py.
    • tools/skill_eval_compare.py, skill_eval_improve.py and skill_eval_pilot.py use the canonical sources.
    • Fingerprinting the canonical files, not the shims, is the right choice.
  • The release boundary is handled openly.
    • The new runtime-identity key makes pre-move plans fail status and recover.
    • A new test, test_pre_move_runtime_identity_is_rejected_without_rewriting_plan, covers this. It asserts the plan bytes are unchanged and no calls are made.
    • The docs add a "Release-boundary recovery" section that says to use the original pinned runtime.
    • An opt-in two-wheel integration test covers recovery. It is skipped unless INFERRSI_PRE_MOVE_PYTHON is set.
  • Cold-import coverage is extended. The cold-import and no-IO test now covers workflows as well as handoff. For workflows it relaxes the infra/benchmark/model block, which is correct because workflows legitimately join those cores.
  • Docs agree with the layout. module-layout.md, architecture.md, function-owners.md and the other touched docs match the new layout.

Not verified here (no execution or repo access)

  • I did not run the test suite or the wheel-install check, and CI status is not seeded.
  • I did not grep the whole tree for leftover inferrsi.apps.campaign / apps.improvement / apps.skill_* references, including patch(...) strings.
  • I did not check that the consumers lists in the inventory are exactly complete. For example, docs/operations.md is listed under campaign, but its diff shows no module path change. tests/test_apps_module_migrations.py should catch any drift.
  • I did not check the wheel package configuration for the new apps/workflows package. The check_installed.py additions would fail if it were missing.

Non-blocking follow-up: the pinned pre-move wheel recovery test is opt-in. Consider running it once in a release or CI job before shipping the version that crosses this boundary.

I am approving as an authorized delegate (Dong-bot). Merge still needs the repo's usual qualifying approvals and green CI.


Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Skill evaluation

PLANNED: not executed; there are no task outcomes yet.
No approval, model efficacy or current-head test result is implied.

  • Next: resolve check applicability at head 940d54ee365769bedefc117530c74ba3eb88345f.
  • Task selection: no supported automatic comparison; choose a relevant source/content check and disposition uncovered claims.
  • Selected comparisons: 0 · Estimated task calls: 0. Uncovered/unsupported changes: 56; Canonical library paths: 0; Generated projection paths: 0; Other paths: 56. These are paths, not independent claims.
  • Return: command, input/source pin, exit status, outputs/failures and untested claims for independent verified/refuted/unverified disposition.
Author verification steps (nonpaid; not yet run)
  1. Bind the checkout. Run git rev-parse HEAD; it must equal 940d54ee365769bedefc117530c74ba3eb88345f. If it differs, obtain a current plan; do not reuse old results.
  2. Choose an applicable check. Match each changed behavior to the task below or a source/content check; a path match alone does not establish relevance.
    No supported automatic comparison is selected. Review the canonical changes and explain which claims need evidence, are unaffected, or remain unverified; do not treat generated records as separate behavioral claims.
    Uncovered scope by location: Canonical library paths: 0; Generated projection paths: 0; Other paths: 56. Path counts are not claim counts; no metadata-only conclusion is inferred.
  3. Return evidence for independent review: head, check/task/source pin, command, environment, exit status, outputs/failures and untested claims. Label scripted controls. Request each claim's disposition: verified, refuted or unverified; this plan proves neither model provenance nor benefit.
    Check recipes and evidence limits

Evidence and logs · Retention: 30 days.
Retained artifacts are verifiable only before Unix time 1797608042 while the linked artifact is available. Expired evidence is unverified.

Plan bindings, budget, frozen exam and advisory details

Advisory: obtain missing evidence

Retained evidence availability: planned / unexecuted. No approval or merge authorization is issued.
Applicable only while these revisions and inputs match the current PR.
Paid execution requires trusted label authorization and cumulative PR reservation. Unknown accounting blocks new paid calls; planning is not execution.
Pipeline evidence only; offline fixed outputs do not establish skill efficacy.
Unchanged-correct is valid neutral evidence, not incremental benefit.
Content/source review and authorship admission remain separate.
Trusted regrading checks retained answers, not model/hardware provenance or a hostile executor; this is not an efficacy certification.

  • Retained evidence is planned / unexecuted.
  • No supported comparison cases were selected.
  • Uncovered/unsupported scope needs a case or explicit reviewer disposition.
    Plan SHA-256: 2ed33bad2676a71bb7161e4e5dfe72d0cfdf9035683b3d9a552b982d8737feb6
    Base: 965ab5cf83399a99530949b04bb934a892ee417c · Head: 940d54ee365769bedefc117530c74ba3eb88345f
    Trusted implementation: 965ab5cf83399a99530949b04bb934a892ee417c

Model/settings: {"effort":"not_applicable","name":"claude-opus-5-5"}
Budget: {"comparisons":0,"estimated_calls":0,"limits_per_comparison":{"campaign_seconds":600,"max_task_calls":2,"repeats":1,"timeout_seconds":180,"token_stop_new_trials":200000},"provider_config_sha256":"b28d80f8194fbe0f5a9f01515efb24b218979fd45f702776560da276f4b30d04","reservation":null}

Skill Comparison Cases Estimated calls

Automatic execution / cumulative PR budget:

{"reason":"Trusted configuration or pricing missing, invalid or expired","remaining_usd":"unknown","reserved_or_unresolved_usd":"unknown","spent_or_legacy_usd":"unknown","state":"blocked"}

Provider/settings: {"client":{"api_version":"2023-06-01","endpoint":"https://yibuapi.com/v1/messages","kind":"anthropic-messages-full-context-v1","max_output_tokens":4096,"max_read_bytes":32768,"max_request_bytes":262144,"max_requests":1,"max_tool_calls":12,"provider":"yibuapi"},"limits":{"token_stop_new_trials":200000},"model":{"effort":"not_applicable","name":"claude-opus-5-5"},"pricing":{"currency":"USD","expires_at":1793271228,"group":"default","max_input_tokens":1000000,"model":"claude-opus-5-5","rate_evidence":"https://github.com/InferMatrixLab/InferRSI/issues/215#issuecomment-6467531341","rates":{"cache_read":"0.4","cache_write":"10","input":"8","output":"40"},"request_fee":"0","unit":"per_million_tokens"},"spend_limit":{"amount":"200","currency":"USD"}}

Uncovered / unsupported paths (56)
  • uncovered: "docs/apps-module-migrations.json"
  • uncovered: "docs/architecture.md"
  • uncovered: "docs/contracts/function-owners.md"
  • uncovered: "docs/harness-improvement.md"
  • uncovered: "docs/knowledge-retrieval-trace.md"
  • uncovered: "docs/local-skill-evaluation.md"
  • uncovered: "docs/module-layout.md"
  • uncovered: "docs/operations.md"
  • uncovered: "docs/optimization-workflow-unification.md"
  • uncovered: "docs/skill-evaluation.md"
  • uncovered: "src/inferrsi/apps/benchmark_operations.py"
  • uncovered: "src/inferrsi/apps/campaign.py"
  • uncovered: "src/inferrsi/apps/cli/init.py"
  • uncovered: "src/inferrsi/apps/improvement.py"
  • uncovered: "src/inferrsi/apps/skill_eval_engineering.py"
  • uncovered: "src/inferrsi/apps/skill_evaluation.py"
  • uncovered: "src/inferrsi/apps/views/capture_delivery.py"
  • uncovered: "src/inferrsi/apps/views/optimization_story.py"
  • uncovered: "src/inferrsi/apps/workflows/init.py"
  • uncovered: "src/inferrsi/apps/workflows/benchmark_operations.py"
  • uncovered: "src/inferrsi/apps/workflows/campaign.py"
  • uncovered: "src/inferrsi/apps/workflows/improvement.py"
  • uncovered: "src/inferrsi/apps/workflows/skill_eval_engineering.py"
  • uncovered: "src/inferrsi/apps/workflows/skill_evaluation.py"
  • uncovered: "src/inferrsi/harness/optimization_session.py"
  • uncovered: "tests/_gate_policy_support.py"
  • uncovered: "tests/check_installed.py"
  • uncovered: "tests/fixtures/dashboard_profile_review.py"
  • uncovered: "tests/fixtures/knowledge_retrieval.py"
  • uncovered: "tests/test_anthropic_agent.py"
  • uncovered: "tests/test_apps_module_migrations.py"
  • uncovered: "tests/test_benchmark_operations.py"
  • uncovered: "tests/test_campaign_record_injection.py"
  • uncovered: "tests/test_campaign_task_selection.py"
  • uncovered: "tests/test_harness_improvement.py"
  • uncovered: "tests/test_knowledge_navigation_browser.py"
  • uncovered: "tests/test_optimization_session.py"
  • uncovered: "tests/test_optimization_story.py"
  • uncovered: "tests/test_package_root_inventory.py"
  • uncovered: "tests/test_project_optimization.py"
  • uncovered: "tests/test_search_archive.py"
  • uncovered: "tests/test_search_attribution.py"
  • uncovered: "tests/test_skill_eval_compare.py"
  • uncovered: "tests/test_skill_eval_engineering.py"
  • uncovered: "tests/test_skill_eval_generation.py"
  • uncovered: "tests/test_skill_eval_local.py"
  • uncovered: "tests/test_skill_eval_local_cli.py"
  • uncovered: "tests/test_skill_eval_pipeline.py"
  • uncovered: "tests/test_skill_evaluation.py"
  • uncovered: "tests/test_target_module_import_gate.py"
  • uncovered: "tools/skill_eval_cloud_replay.py"
  • uncovered: "tools/skill_eval_compare.py"
  • uncovered: "tools/skill_eval_improve.py"
  • uncovered: "tools/skill_eval_local.py"
  • uncovered: "tools/skill_eval_pilot.py"
  • uncovered: "tools/skill_eval_qualify.py"

@Dong1017 Dong1017 added kind:task Bounded delivery task with a delegated slice area:harness Harness module: profile, A/B loop, quality gates, evidence reports labels Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

补充 Dong 检视中未执行部分的作者证据(head 940d54ee365769bedefc117530c74ba3eb88345f):

  • 已扫描 src/tests/tools/plugins 的绝对、相对导入,以及测试补丁和 controller/runtime 路径字符串。test_active_consumers_use_canonical_paths_and_are_registered 对活跃静态导入强制 canonical 路径,并要求登记;最终清单测试 24 passed。旧路径的保留项是 shim、兼容回归和历史复现材料。
  • consumers 同时登记导入和操作/字符串约束:docs/operations.md 关联的是新增的固定 runtime 恢复说明;Harness 的 campaign 摘要虽为文件字符串而非导入,也已登记。
  • 已实际运行 test_pinned_pre_move_wheel_recovers_its_original_plan,两个解释器分别仅安装 base 与 candidate 的 wheel,在仓库外以 -I 启动,1 passed;并非使用 patched identity 放行。check_installed.py 通过,资源清单 930 项一致。
  • CI 的 Python 3.10/3.11 installed-package 和 pr-references 已通过,其他 Ready jobs 继续运行。扩展回归的 6 failed / 2 errors 均已在 base 原码复现,PR 中列出的环境限制保持公开。

Copy link
Copy Markdown
Contributor Author

当前 head 未变。Ready CI 已有七项通过:Python 3.10/3.11 full-check、两套 installed-package、dashboard-browser、Windows host-compatibility 和 pr-references;knowledge-model-policy 为 success / 不适用。

macOS host-compatibility 排队后被取消,不计通过;已仅重跑这一 job。当前仍只有 Dong1017 对当前 head 的一份有效独立审批,继续等待第二份有效审批及 macOS 结果,满足条件后 squash。第四步 integrations 已在独立工作区预备,284 项回归及 332 个子用例通过,待当前切片合入后另提 Ready PR。

Copy link
Copy Markdown
Contributor Author

macOS 补跑也在等待约 15 分钟后取消。GitHub 检查记录明确提示 hosted runner 未被分配,并提示 macOS arm64 容量受限;该 job 的 steps 为空。记录见 取消的 macOS job。其余七项检查在最新 attempt 仍均为 success。

保持现有 CI 要求,不把取消计为通过,稍后退避补跑。当前正式有效审批仍为 Dong1017 一份;第二份审批和实际完成的 macOS 检查仍是 #504 的合入前置。第四步分支已推送,比较与验证摘要已回填 #490。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:harness Harness module: profile, A/B loop, quality gates, evidence reports kind:task Bounded delivery task with a delegated slice

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants