Skip to content

refactor(benchmark): move the metric contract and validators out of model - #503

Open
Dong1017 wants to merge 2 commits into
mainfrom
impl/498-s1-benchmark
Open

Dong1017 wants to merge 2 commits into
mainfrom
impl/498-s1-benchmark

Conversation

@Dong1017

@Dong1017 Dong1017 commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Governing Issue and delivery slice

Refs #498

  • Accepted scope / delegation: slice 1 of task: retire the infra/ and model/ packages into their owning modules #498, confirmed by the issue owner in the plan refresh and the issue body. The body records the slice and its checklist.
  • What this PR delivers:
    • Moves, with no forwarder or alias:
      • inferrsi.model.protocols.metrics becomes inferrsi.benchmark.metrics.
      • inferrsi.model.validators (flare, image) becomes inferrsi.benchmark.validators.
    • Benchmark's own helpers: benchmark/artifacts.py holds sha256, write_json and atomic_write_json.
    • Switched consumers: the legacy run gates and the MoE task program.
    • Retirement guards: a gate check and an installed-wheel check for the retired paths.
    • Tests and docs: the matching tests, docs and CHANGELOG.
  • What it does not deliver: slices 2–5. inferrsi.model.protocols.image, the provider modules and all of inferrsi.infra stay, and so does infra/artifacts.py for its other users.

Knowledge authorship declaration

No path protected by docs/knowledge-model-policy.md changes. The three Knowledge files that cite src/inferrsi/model/protocols/metrics.py pin commit b3ef95a and are unchanged. The declaration below is disclosure only.

{"schema_version":1,"head_sha":"b508bd32c39c0dbf373f26b8a210c57fd49e8b92","human_only":false,"material_ai_authors":[{"tool":"Claude Code","model_id":"claude-opus-5-5","effort":"xhigh","role":"author"}]}

Changed artifacts and evidence

Requirement (#498 slice 1) Actual change Check / evidence and result Remaining gap
Metric contract and validators move to Benchmark git mv to benchmark/metrics.py and benchmark/validators/. Only the import lines change in the moved files. test_legacy_run_split.py (Benchmark rules import only inferrsi.benchmark.metrics), test_flare_adapter.py, test_image_modules.py, test_diffusion_reference_psnr.py, test_moe_benchmark.py: pass on H200 (see Checks) none
Benchmark copy of the artifact helpers, with a drift guard benchmark/artifacts.py; new tests/test_artifact_helper_copies.py checks that every copy (Infra, Harness, Benchmark) has the same function source, retry constants and written bytes, and that the Benchmark copy imports no other inferrsi module as above Slice 5 drops infra from the copy table. A later slice adds the Apps copy.
Old paths fail even with no file behind them RETIRED_MODULES and _retired_imports in tests/_gate_policy_support.py. New negative controls: absolute, relative, from … import metrics, from inferrsi.model import validators and a returning file. Three more controls import the old paths relatively from a package initializer (b508bd3, after review). A control shows inferrsi.model.protocols.image is not retired. test_target_module_import_gate.py: pass on H200 (see Checks) none
No forwarder in the installed wheel tests/check_installed.py: find_spec is None for both old paths, the four new modules ship, -m inferrsi.benchmark.validators.flare --help exits 0, and the old -m path exits 1 with No module named 'inferrsi.model.validators' installed-package job on this head: pending pending CI
Flare CLI and receipts Tests use the new command. test_the_old_flare_cli_path_is_gone checks the old command. test_new_receipts_bind_the_benchmark_adapter_files checks that a new receipt keeps its schema and binds benchmark/validators/flare.py and benchmark/artifacts.py. test_flare_adapter.py: pass on H200 (see Checks) none
Docs and CHANGELOG docs/flare-qualification.md, docs/operations.md, docs/architecture.md, docs/contributing.md, docs/module-layout.md, src/inferrsi/model/README.md, CHANGELOG git grep for the old paths finds only intended mentions: CHANGELOG, the layout table, the historical docs/reviews/source-audit-ac70272.md, the three Knowledge files pinned at b3ef95a, and the retirement tests none

Checks and dependencies

  • Checks actually run:
    • H200 Linux, Python 3.12.13, the tree committed as f8f1553. I ran a git archive of the commit. All 2706 files matched a SHA-256 manifest before the run. The selected files were:

      • the 15 files test_target_module_import_gate.py, test_artifact_helper_copies.py, test_flare_adapter.py, test_legacy_run_split.py, test_module_boundaries.py, test_moe_benchmark.py, test_image_modules.py, test_diffusion_reference_psnr.py, test_architecture_closed_loop.py, test_first_wave_isolation_policy.py, test_knowledge_compatibility_shells.py, test_legacy_run_forwarder_gate.py, test_campaign_record_injection.py, test_package_root_inventory.py and test_knowledge_model_policy.py;
      • plus test_skill_evaluation.py::test_harness_artifact_primitives_match_infra_bytes_and_semantics.

      Result: 510 passed, 1 skipped, 384 subtests passed. The skip is test_package_root_inventory.py:153, which needs Git metadata that the archive does not have. full-check runs it.

    • Two H200 environment problems, fixed outside the tree. Two earlier attempts on the same verified tree failed for these reasons, not for code reasons:

      • H200's system site has cufile-python, which installs a regular top-level package named tests. That package shadows the repository's namespace tests, so 8 files failed to collect. A shim directory outside the tree points tests back at the repository.
      • A relative PYTHONPATH=src was not resolvable from the MoE launcher's child working directory, so test_frozen_launcher_uses_generic_executor_and_fake_torchrun could not import inferrsi. An absolute path fixed it.
    • Lint: Ruff 0.17.5 on src tests tools and git diff --check pass at f8f1553 and b508bd3.

    • full-check at f8f1553 (run 44331108229): 3.10 and 3.11 each 5215 passed, 37 skipped, 672 subtests, on GitHub's merge tree with main 769522f. Every other check on that head also succeeded.

    • b508bd3 adds only three gate controls in tests/test_target_module_import_gate.py. Locally, -k retired in that file gives 15 passed. The ready lane on b508bd3 is the full-suite proof for it.

    • Local Windows: the same targeted run on the local laptop was stopped for low system memory and gave no result.

  • Failed, skipped or not run:
    • check_installed.py needs a built wheel, so installed-package on this PR is its proof.
    • The full suite runs in full-check on this PR.
    • No NPU run. The MoE change is two import lines, and the MoE no-device contract tests cover it.
  • Compatibility:
    • The Flare command path changes. Arguments, outputs and exit codes (0 validated, 2 rejected) do not.
    • New Flare receipts carry new adapter digests. No receipt is stored in the repository, and the adapter reads packets, not earlier receipts.
    • Pickled objects that name an old path no longer load.
    • Each MoE evaluate call prepares and launches its own copy of the task files in one call, so no directory prepared before this change is launched again.
    • Optimization sessions are not affected: harness/optimization_session.py::_runtime_identity() names no file this PR changes.
  • Dependencies: none. Slices 2–5 are independent of this PR, apart from slice 5's infra removal from the copy table.

Remaining work and Issue closure

Provenance

The code, tests and docs were written in a Claude Code session reported as claude-opus-5-5, effort xhigh, as confirmed by the owner. This is client and session attribution, not a backend attestation. Each commit has DCO sign-off and an Agent-assisted-by line.

🤖 Generated with Claude Code

…odel

The finite positive metric contract and the Flare and image quality checks
are Benchmark rules, so they move to Benchmark with no forwarder or alias:

- model/protocols/metrics.py -> benchmark/metrics.py
- model/validators/{__init__,flare,image}.py -> benchmark/validators/

benchmark/artifacts.py holds Benchmark's own sha256, write_json and
atomic_write_json, with the same function source as the Infra and Harness
copies, so Benchmark imports no Infra module for them. The new
tests/test_artifact_helper_copies.py keeps every copy equal in source and
written bytes. infra/artifacts.py stays for its other users.

The legacy run gates and the MoE task program import the Benchmark
modules. The Flare command is now python -m
inferrsi.benchmark.validators.flare with the same arguments, outputs and
exit codes. A new Flare receipt keeps its schema and keys, and its adapter
digests name the moved files.

The gate now fails a file under a retired path and any import that names
one, even when no file is behind it. The installed-wheel check asserts that
the old paths are absent, the new ones ship, and only the new Flare command
runs. Docs and CHANGELOG name the new paths. Knowledge records that pin
src/inferrsi/model/protocols/metrics.py at an earlier commit keep that path.

Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author)
Signed-off-by: GUOGUO <xwdong1998@163.com>

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent initial review of #498 slice 1: base 769522f → head f8f1553. No prior review from this account on this PR.

Source conclusion: no definite blocking code defect found in this bounded move. This is COMMENT while necessary current-head checks remain incomplete, not acceptance of the entire #498 retirement.

What I independently verified:

  • Compared the actual before/after contents: metrics.py and validators/init.py are byte-identical moves; flare.py and image.py differ only in the expected artifact-helper import lines.
  • Benchmark's sha256/write_json/atomic_write_json implementations match the existing copies, including bounded replace retries and nonfinite-JSON rejection; the new parity tests check source, constants and emitted bytes. No cross-core import is introduced by the new helper.
  • Legacy-run and MoE imports use the canonical Benchmark modules. Flare still derives new receipt adapter hashes from the actual imported files; its receipt schema remains unchanged. The new receipt test checks that binding rather than rewriting old receipt evidence.
  • The retired-path guard inspects spelled absolute and relative imports even when the retired target no longer exists, and rejects returning modules beneath the retired prefixes. The unretired image request protocol remains outside those prefixes.
  • Installed-wheel checks verify absence of old modules and presence of canonical modules, and invoke new/old Flare CLI paths in an isolated interpreter from a temporary directory. Matching tests and docs explicitly disclose the CLI/pickle break instead of adding aliases.
  • Reviewed all 25 changed-file diffs, relevant source implementations, AGENTS and current #498 slice-1 requirements. Slices 2–5 and their owner prerequisites are not delivered or approved by this review.

Execution evidence actually inspected:

Before approval: obtain successful necessary current-head full/host/declaration results and inspect any failure or relevant skip. No new source fix is requested by this round. Hardware performance is not established by a path migration or CPU smoke; no new NPU claim is made.

Non-blocking suggestion: add a retired-module relative-import negative control inside a package init.py, in addition to the ordinary-module relative-import controls.

No repository code, tests, builds or devices were executed by me; no other reviewers' opinions were read; no files, branches, Issue state, merge or auto-merge settings were changed.

@Dong1017 Dong1017 added the kind:task Bounded delivery task with a delegated slice label Nov 18, 2026
Review of f8f1553: the retired-path controls used ordinary modules only.
A package initializer resolves a relative import from its own package, so
three new controls import the retired metric contract and validators
relatively from benchmark/__init__.py and benchmark/sub/__init__.py, and
expect the violation under the package name.

Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author)
Signed-off-by: GUOGUO <xwdong1998@163.com>
zengchuang-hw
zengchuang-hw previously approved these changes Nov 18, 2026

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent evidence follow-up to #503 (review). Reviewed range remains base 769522f → head f8f1553; no code delta since my preceding review.

Approve this bounded #498 slice-1 migration. My independent 25-file/source review found no blocking defect: metric and validator moves preserve implementation, canonical consumers and installed-wheel checks migrate together, artifact helper copies preserve bytes/retry behavior, receipt hashes follow the actual new adapter, and the retired-path gate rejects old absolute/relative imports without alias shims. This is not approval of #498 slices 2–5 or a claim of new hardware performance.

The outstanding execution-evidence condition is now resolved. I read original logs, not merely green check badges:

I inspected the skip list: it includes optional numpy image/PSNR numerical tests, optional torch/cannbot/device paths, platform-specific and browser tests. I do not claim those skipped paths ran. The image validator is an import-only move with its numerical implementation independently compared unchanged; wheel import/CLI checks and helper parity tests cover the changed path/helper contract. Thus those optional numerical skips are not a blocker for this bounded migration. The separate browser job passed; no real NPU/model experiment was executed or inferred.

Resolved: the prior pending full/host/declaration evidence condition. No required source correction remains from my review. The relative-import negative test inside package init.py remains a non-blocking suggestion.

Rechecked author, repository ID, current head and my prior reviews immediately before submission. No repository code, tests, builds or device jobs were executed by me; no other reviewers' opinions were used; no merge or auto-merge action was taken.

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent incremental review: f8f1553 → b508bd3. Previous review: #503 (review) (bound to the old head; now dismissed after this update).

The single-file, 16-line change addresses my non-blocking test suggestion. I independently traced all three fixtures through _module_name, graph package identity, _from_base and _retired_imports: the direct Benchmark initializer resolves '..model.protocols' from inferrsi.benchmark, and nested initializers resolve '...model.validators' / '...model' from inferrsi.benchmark.sub. The expected retired targets and package-name diagnostics are correct, without requiring the retired modules to exist. Production logic and the previously reviewed migration are unchanged. No new source defect found.

Resolved in source: the initializer-relative negative controls requested previously are now present for metrics, a validator child module and the validator package. No new non-blocking suggestion.

Evidence: inspected the fresh lint-and-smoke original log (Ruff passes; 8 passed in 0.04s): https://github.com/InferMatrixLab/InferRSI/actions/runs/44335577257/job/134911232127 . This smoke does not prove the newly added parameterized gate tests executed. Current installed-package 3.10/3.11, Windows, references and declarations report success; full-check 3.10/3.11 are still in progress, macOS is queued and browser is in progress. The earlier full results belong to f8f1553, not this head.

Before approval: successful necessary current-head regression evidence, including the three new initializer controls, and completion of remaining required checks. No code correction is requested by this round. COMMENT is intentional while that evidence is pending, rather than carrying the old approval to a new SHA.

Read actual diff and relevant test/gate implementation, reused the unchanged AGENTS and governing slice-1 scope from my initial review, and rechecked current head and my prior submissions. No repository code/tests/builds were executed, no other reviewers' opinions read, no merge or auto-merge.

@Dong1017

Copy link
Copy Markdown
Collaborator Author

New head: b508bd32c39c0dbf373f26b8a210c57fd49e8b92.

This push voided the approval given at f8f1553 (review 5959576659). I pushed it after that approval had been submitted. The change is test-only and is the non-blocking suggestion from the first review:

  • tests/test_target_module_import_gate.py: three controls import the retired metric contract and validators relatively from benchmark/__init__.py and benchmark/sub/__init__.py, and expect the violation under the package name.
  • No source, doc or other test file changed between f8f1553 and b508bd3.

Evidence:

  • Locally, -k retired in that file gives 15 passed.
  • Ruff 0.17.5 and git diff --check pass.
  • full-check at f8f1553 gave 5215 passed, 37 skipped and 672 subtests on 3.10 and 3.11 (run 44331108229).
  • The ready lane on b508bd3 is the full-suite proof for the new controls.

The PR body's head_sha and evidence are updated.

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent evidence follow-up to #503 (review). Code-review range remains f8f1553 → b508bd3; no additional PR code change since that review.

Approve this exact head. The new three initializer-relative import controls correctly exercise package-relative resolution and exact retired-path diagnostics, addressing my earlier optional suggestion without changing production behavior. The original bounded slice-1 migration review remains applicable. No blocking source issue remains, and the previously pending fresh-head checks have now completed.

I inspected original current-run logs:

These logs identify synthetic merge 1c4d6c1, merging this head into 965ab5c. This is distinct from the old run's integration base 769522f; I do not attribute the entire test-count increase solely to this PR. Full pytest ran without failure; the new parameterized gate tests have no skip/dependency condition. The optional image/PSNR numerical skips remain disclosed as in my prior review and do not imply those paths ran. Current installed-package, browser, references and declarations checks also report success.

Resolved: current-head regression/host evidence gap. No required source correction or further non-blocking suggestion remains from this incremental review. Approval is limited to this PR, not #498's later slices or new hardware qualification.

Rechecked exact head, author, repository ID and my existing reviews before submitting. No repository code, tests, builds or devices executed by me, no other reviewers' opinions used, and no merge or auto-merge action.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind:task Bounded delivery task with a delegated slice

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants