Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,31 @@

### Changed

- The metric contract and the quality validators move from `inferrsi.model` to
Benchmark (Refs #498, slice 1). There is no forwarding module or alias, and the
import gate fails any import of an old path, even when no file is behind it.
- `inferrsi.model.protocols.metrics` is now `inferrsi.benchmark.metrics`.
`inferrsi.model.validators` (`flare`, `image`) is now
`inferrsi.benchmark.validators`.
- The Flare adapter command is now
`python -m inferrsi.benchmark.validators.flare`, with the same arguments,
outputs and exit codes. The old `python -m inferrsi.model.validators.flare`
fails with `ModuleNotFoundError`.
- New `inferrsi.benchmark.artifacts` holds Benchmark's own `sha256`,
`write_json` and `atomic_write_json`. Their code equals the copies in
`inferrsi.infra.artifacts` and Harness, and
`tests/test_artifact_helper_copies.py` keeps the copies equal.
`inferrsi.infra.artifacts` stays for its other users.
- A new Flare receipt keeps its schema and keys. Its `adapter` digests are those
of `benchmark/validators/flare.py` and `benchmark/artifacts.py`. Existing
receipts are not rewritten.
- The MoE task's `benchmark.py` imports the Benchmark modules. Each MoE
`evaluate` call prepares its own copy of the task files and launches that copy
in the same call, so a new run binds the changed `benchmark.py` bytes and no
directory prepared before this change is launched again.
- Pickled objects that name an old module path no longer load. The Knowledge
records that cite `src/inferrsi/model/protocols/metrics.py` at commit
`b3ef95a` keep that pinned path.
- The five runtime resource loaders are closed and the combined tree is isolated
(Refs #412, R3). `first_wave_report()` now has no violation and no blocker, and
`new_core_closure_isolated` and `isolation_complete` are both true.
Expand Down
16 changes: 9 additions & 7 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ interfaces; existing CLI commands remain supported.
| --- | --- | --- |
| Entry and knowledge | CLI arguments, immutable RunSpec, cited knowledge/context export | Model execution or automatic source editing |
| Harness | AB/BA order, deadlines, source checks, scenario composition, evidence, performance adjudication and reporting | Device-specific conditions, request schemas or kernel dispatch |
| Model | Request payloads, metric schemas, output decoding and explicit quality checks | Scheduling, worker lifetime or hardware selection |
| Model | Request payloads and output decoding | Scheduling, worker lifetime, hardware selection, metric contracts or quality checks |
| Infra | Repository identity, device discovery/qualification, process ownership, HTTP transport, storage and artifacts | Model constants, campaign gates or performance verdicts |

Dependencies flow from entrypoints to Harness, from scenarios to Model/Infra,
Expand Down Expand Up @@ -163,6 +163,11 @@ src/inferrsi/
registry.py Generic task resource discovery
tasks/ Task-specific definitions, workloads and assessment
legacy_run_v0.py Old run gates: metric contract, summaries and verdict
metrics.py Finite positive dotted-key metric contract
artifacts.py Benchmark-owned file digests and JSON writes
validators/
flare.py Offline Flare delivery-evidence adapter and pair check
image.py Explicit PNG dimensions/mode, hashes, RGB/alpha quality gates
sampling.py Warmup seeds, first-response retention, sample/artifact groups
skill_evaluation.py Skill trial check validity, verdict and condition statistics
skill_eval_diagnostic.py Synthetic KV admission diagnosis task and grader
Expand Down Expand Up @@ -258,10 +263,7 @@ src/inferrsi/
external.py External command trials; metric validation injected
model/
protocols/
metrics.py Finite positive dotted-key metric contract
image.py OpenAI-style image payload and PNG response protocol
validators/
image.py Explicit PNG dimensions/mode, hashes, RGB/alpha quality gates
infra/
processes.py Supervised Linux serving workers and their group cleanup
targets/git.py Repository identity and source snapshots
Expand Down Expand Up @@ -301,7 +303,7 @@ path to its replacement and lists what still keeps the isolation gate false.

| Add | Main extension locations | Required qualification |
| --- | --- | --- |
| Model or scenario | `model/protocols`, `model/validators`, `harness/scenarios` | Frozen request and metric contract; model-specific quality coverage |
| Model or scenario | `model/protocols`, `benchmark/metrics.py`, `benchmark/validators`, `harness/scenarios` | Frozen request and metric contract; model-specific quality coverage |
| Repository | `infra/targets`, a composed scenario, `data/repo_packs` | Stable source identity; command/result contract; environment isolation |
| Hardware | `infra/platforms` | Device selection, process exclusivity and independent backend/model evidence; a backend is registered in `infra/platforms/__init__.py` and its probe command is defined in the implementing module under `infra/platforms/`, not in package data |

Expand All @@ -326,7 +328,7 @@ benchmark format, reuse the standard executor and only replace metric interpreta
from functools import partial

from inferrsi.apps.legacy.run import execute_trial, run_experiment
from inferrsi.model.protocols.metrics import read_metric
from inferrsi.benchmark.metrics import read_metric


def read_engine_metric(document, key):
Expand All @@ -346,7 +348,7 @@ cleanup. A custom `execute(command, cwd, trial_dir, deadline, metric_key)` must
preserve that trial record contract. Prefer composing `execute_trial` over
reimplementing process lifetime. Metric readers convert a result document into a
float; `benchmark.legacy_run_v0.trial_metric` then applies the one finite, positive
metric contract (`model.protocols.metrics.validate_metric`) to whatever the reader
metric contract (`benchmark.metrics.validate_metric`) to whatever the reader
returns, so a reader that yields `0.0`, a `bool` or a non-finite value records a
metric error rather than entering the evaluation. The Harness executor
`harness.scenarios.external.execute_trial` holds no metric rule: a caller that
Expand Down
2 changes: 1 addition & 1 deletion docs/contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ policy does not renormalize existing blobs or authorize edits to frozen files.
Nightly checks cover Linux and Windows checkouts with `core.autocrlf` set to
`false`, `true` and `input`.

Use `harness/scenarios` and `model/protocols` / `model/validators` for new models,
Use `harness/scenarios`, `model/protocols` and `benchmark/validators` for new models,
`infra/targets` with `data/repo_packs` for repositories, and `infra/platforms` for
hardware. A backend is registered in `infra/platforms/__init__.py`; its probe
command is defined in the implementing module under `infra/platforms/`. That
Expand Down
10 changes: 5 additions & 5 deletions docs/flare-qualification.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

The adapter reads existing Flare qualification artifacts and writes a separate
receipt. It never launches Flare, downloads a model or executes imported code.
Run it with `python -m inferrsi.model.validators.flare`. The existing
Run it with `python -m inferrsi.benchmark.validators.flare`. The existing
`external_benchmark` scenario, `validation_command` and evaluator own execution
and verdicts.

Expand Down Expand Up @@ -92,19 +92,19 @@ synthetic status, package bytes, corpus bytes and prepared capacity. This keeps
an incomplete or incomparable pair out of the evaluator's performance inputs.

```text
python -m inferrsi.model.validators.flare measure \
python -m inferrsi.benchmark.validators.flare measure \
--packet <baseline>/packet.json --peer <candidate>/packet.json \
--output <receipts>/baseline.json

python -m inferrsi.model.validators.flare validate-pair \
python -m inferrsi.benchmark.validators.flare validate-pair \
--baseline <baseline>/packet.json --candidate <candidate>/packet.json \
--output <receipts>/pair.json
```

For `RunSpec`, use these argv sequences:

```python
cli = (python, "-m", "inferrsi.model.validators.flare")
cli = (python, "-m", "inferrsi.benchmark.validators.flare")
baseline_command = cli + (
"measure", "--packet", baseline, "--peer", candidate, "--output", "{result}",
)
Expand Down Expand Up @@ -145,7 +145,7 @@ From a development checkout with InferRSI installed:

```text
python tests/fixtures/flare_packet.py <sample-directory>
python -m inferrsi.model.validators.flare measure \
python -m inferrsi.benchmark.validators.flare measure \
--packet <sample-directory>/packet.json --peer <sample-directory>/packet.json \
--output <outside-sample-directory>/receipt.json
pytest tests/test_flare_adapter.py
Expand Down
4 changes: 2 additions & 2 deletions docs/module-layout.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,10 +145,10 @@ Several flat modules remain shared seams until their slice starts.
| `apps/mindie/` (`project.py`, `context.py`, `mcp.py`, `read_worker.py`) | The local MindIE adapter: project grant, stdio MCP, context export and the read worker | M1 (#452) moved the four root MindIE files here; no Harness, Knowledge or Infra module names MindIE, and the initializer imports nothing |
| `harness/` | Mixed legacy tree, not target isolation | Not redefined |
| `harness/runner.py`, `harness/scenarios/external.py`, `harness/atomic_json.py` | Old run execution facts and Harness-owned JSON writes | Ports injected; no Model edge, and no Infra edge except the local process executor |
| `benchmark/legacy_run_v0.py`, `benchmark/sampling.py` | Old run gates and metric validation; image/video sample collection | Imports only `model/protocols/metrics`; owns its rule inputs and verdict labels |
| `benchmark/legacy_run_v0.py`, `benchmark/sampling.py` | Old run gates and metric validation; image/video sample collection | Imports only `benchmark/metrics.py`; owns its rule inputs and verdict labels |
| `apps/legacy/` | Old run composition, old `evaluate` entry and HTML report | Supplies the Git source snapshot from `infra/targets`; remains in Apps |
| `apps/inspection.py` | Prepared-environment inspection bound to this host | Binds the `infra/inspection.py` facts API to `infra.commands.probe_command` and `infra.targets.repository_identity`; replaces root `inspection.py`, which R1 deleted with the three old-run root forwarders |
| `model/validators/` | Benchmark-leaning checks | Not moved |
| `benchmark/metrics.py`, `benchmark/validators/`, `benchmark/artifacts.py` | The metric contract, the Flare and image quality checks, and Benchmark's own file digests and JSON writes | #498 slice 1 moved the first two from `model/protocols/metrics.py` and `model/validators/`, with no forwarder or alias; the gate fails any import of the old paths |
| `infra/` | Seams used by the current tree, including the `infra/inspection.py` facts API | Not a fourth core; reaches no Knowledge, Harness, Benchmark or Apps module |
| `apps/views/summary_chart.py` | Stdlib SVG gain chart and report figure spec | A2 (#408) moved root `summary_chart.py` here; the renderer, its specification checks and its `main` are unchanged, and `tools/campaign_summary_chart.py` calls it |
| `contracts/` (`runspec.py`, `json_v1.py`, `claim.py`, `evidence.py`) | Technical-only contracts: RunSpec, canonical JSON, verdict vocabulary, evidence segment and path rules | Exact module set; the initializer re-exports only `RunSpec`; not a fourth core |
Expand Down
2 changes: 1 addition & 1 deletion docs/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ Effect classes used below:
| Assemble an optimization trace | CLI `campaign assemble-trace`; [offline dashboard](dashboard.md#capture-the-optimization-trace) | Local evidence-writing | Copies an explicitly reviewed context into a new bounded campaign-root snapshot; caller owns redaction and narrative accuracy |
| Import a MindIE read | CLI `campaign import-mindie`; [Codex workflow](codex-plugin.md#mindie-knowledge) | Local evidence-writing | Retains caller-supplied query/explain bytes and hashes, exports delivered graded bodies and a trace context fragment; no activation, model call, transcript capture or automatic caller review |
| Associate runs with a local campaign | CLI `campaign init` / `campaign record` / `campaign decide`; [offline dashboard](dashboard.md) | Local evidence-writing | Legacy manual records reference existing artifacts and never start an agent; record/decide cannot mutate a managed optimization campaign |
| Adapt existing Flare delivery evidence | [Offline qualification adapter](flare-qualification.md), `python -m inferrsi.model.validators.flare` | Read-only | Existing raw packets and a separate receipt; one HTTP delivery metric, no model/server/device execution |
| Adapt existing Flare delivery evidence | [Offline qualification adapter](flare-qualification.md), `python -m inferrsi.benchmark.validators.flare` | Read-only | Existing raw packets and a separate receipt; one HTTP delivery metric, no model/server/device execution |
| Evaluate selected skill revisions | [Bounded skill-evaluation driver](skill-evaluation.md) | Experiment-executing | Explicitly authorized real Codex calls, independent task verdicts and private evidence; separate from numeric engine-performance scoring |
| Compare skill additions or updates | [Configured skill comparison](skill-eval-comparison.md), `python -m tools.skill_eval_compare` | Read-only preparation / experiment-executing / report-generating | Fixed Git snapshots and one shared exam; explicit plan authorization for execution; JSON + Markdown regeneration makes no model calls |
| Export a cloud-generated parser pair | [Generation-only protocol](skill-eval-generation-only.md), `python -m tools.skill_eval_cloud_export` | Local evidence-writing; no execution or publication | Validates frozen inputs and retained sources; writes one bounded inert JSON bundle, never executes parser source |
Expand Down
51 changes: 51 additions & 0 deletions src/inferrsi/benchmark/artifacts.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# Benchmark-owned file digests and JSON evidence writes. The function bodies equal
# those in the other cores' copies; tests/test_artifact_helper_copies.py keeps them
# equal. Benchmark imports no other inferrsi module here.

import hashlib
import json
import os
import time
import uuid
from pathlib import Path

# Windows refuses os.replace while another handle holds the destination open.
# A bounded retry covers the observed contention without hiding a real failure.
REPLACE_ATTEMPTS = 8
REPLACE_DELAY_SECONDS = 0.05


def sha256(path):
hasher = hashlib.sha256()
with Path(path).open("rb") as stream:
for block in iter(lambda: stream.read(1024 * 1024), b""):
hasher.update(block)
return hasher.hexdigest()


def write_json(path, value):
Path(path).parent.mkdir(parents=True, exist_ok=True)
Path(path).write_text(
json.dumps(value, ensure_ascii=False, indent=2, allow_nan=False) + "\n", encoding="utf-8"
)


def atomic_write_json(path: Path, value: object) -> None:
path = Path(path)
# A per-writer temporary name keeps two writers from sharing one file.
temporary = path.with_name(f"{path.name}.{os.getpid()}.{uuid.uuid4().hex[:8]}.tmp")
temporary.write_text(
json.dumps(value, ensure_ascii=False, indent=2, allow_nan=False) + "\n", encoding="utf-8"
)
try:
for attempt in range(REPLACE_ATTEMPTS):
try:
temporary.replace(path)
return
except PermissionError:
if attempt + 1 == REPLACE_ATTEMPTS:
raise
time.sleep(REPLACE_DELAY_SECONDS)
finally:
if temporary.exists():
temporary.unlink()
4 changes: 2 additions & 2 deletions src/inferrsi/benchmark/legacy_run_v0.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@
from collections.abc import Callable
from dataclasses import dataclass

from ..model.protocols.metrics import read_metric as read_metric
from ..model.protocols.metrics import validate_metric
from .metrics import read_metric as read_metric
from .metrics import validate_metric

# The v0 protocol's own labels. Benchmark shares no verdict type with another module.
PASS, FAIL, INCONCLUSIVE = "PASS", "FAIL", "INCONCLUSIVE"
Expand Down
4 changes: 2 additions & 2 deletions src/inferrsi/benchmark/tasks/moe-forward-npu/benchmark.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@
import uuid
from pathlib import Path

from inferrsi.infra.artifacts import atomic_write_json
from inferrsi.model.protocols.metrics import read_metric
from inferrsi.benchmark.artifacts import atomic_write_json
from inferrsi.benchmark.metrics import read_metric

# Worker cleanup for this MoE invocation, including detached torchrun sessions.
# torchrun workers may create their own sessions, outside the executor's process
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
from decimal import Decimal, InvalidOperation
from pathlib import Path

from ...infra import artifacts
from .. import artifacts

SOURCE = {
"repository": "https://github.com/NVIDIA/flare-local-inference",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
import math
from dataclasses import dataclass

from ...infra.artifacts import sha256
from ..artifacts import sha256


@dataclass(frozen=True)
Expand Down
7 changes: 4 additions & 3 deletions src/inferrsi/model/README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
# Model

`protocols/` defines request and metric contracts. `validators/` owns output
quality checks with explicit campaign-supplied gates. The image validator's
optional dependencies are available through `inferrsi[image]`.
`protocols/` defines request contracts. The metric contract and the output
quality checks moved to `inferrsi.benchmark.metrics` and
`inferrsi.benchmark.validators` (#498 slice 1). The image validator's optional
dependencies are available through `inferrsi[image]`.

Model/revision pins and numeric acceptance gates stay in campaign plans. Frozen
historical snapshots remain in `tests/fixtures/legacy/`; current examples and
Expand Down
31 changes: 31 additions & 0 deletions tests/_gate_policy_support.py
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,11 @@
}
)
RETIRED_ROOT_MODULES = RETIRED_BY_R1 | RETIRED_BY_A1 | RETIRED_BY_A2 | RETIRED_BY_M1
# #498 slice 1 moved these packages and modules out of ``inferrsi.model``, with no
# forwarder or alias: ``inferrsi.benchmark.metrics`` and ``inferrsi.benchmark.validators``.
# Each name covers everything below it. A file under one fails, and so does any import
# that names one, even when its target is absent.
RETIRED_MODULES = frozenset({"inferrsi.model.protocols.metrics", "inferrsi.model.validators"})
# #412 R1, permanent: the one importer of the neutral data leaf outside
# ``inferrsi.knowledge``, with the only name it may bind (D020: Infra reads one pack
# description and reaches no Knowledge or Harness). This is not a manifest
Expand Down Expand Up @@ -153,6 +158,22 @@ def _from_base(node: ast.ImportFrom, package: str) -> str:
return ".".join(keep + ([node.module] if node.module else []))


def _retired_imports(tree: ast.Module, package: str) -> set[str]:
"""Every name under ``RETIRED_MODULES`` that an import statement in ``tree`` spells."""
found: set[str] = set()
for node in ast.walk(tree):
if isinstance(node, ast.Import):
names = [alias.name for alias in node.names]
elif isinstance(node, ast.ImportFrom):
base = _from_base(node, package)
names = [base, *(f"{base}.{alias.name}" for alias in node.names)]
else:
continue
found.update(
name for name in names if any(_matches(name, old) for old in RETIRED_MODULES))
return found


def _is_entry_delegation(node: ast.stmt, package: str) -> bool:
"""The module entry's one admitted statement, in its exact plain shape."""
module, name = MODULE_ENTRY_DELEGATION
Expand Down Expand Up @@ -729,6 +750,16 @@ def first_wave_report(
for name in modules
if name in RETIRED_ROOT_MODULES and name not in sources
]
violations += [
f"{name} is a retired module"
for name in modules
if any(_matches(name, old) for old in RETIRED_MODULES)
]
violations += [
f"{name} imports retired module {target}"
for name in modules
for target in sorted(_retired_imports(graph.trees[name], graph.packages[name]))
]
# Outside Knowledge the leaf has its one permanent importer and any staged pair
# that names it. Every other importer fails.
declared_importers = dict(LEAF_IMPORTERS)
Expand Down
Loading
Loading