Skip to content
40 changes: 40 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,46 @@

### Changed

- The five runtime resource loaders are closed and the combined tree is isolated
(Refs #412, R3). `first_wave_report()` now has no violation and no blocker, and
`new_core_closure_isolated` and `isolation_complete` are both true.
- The Harness task protocol no longer runs in the Harness process. Each
`validate_arguments`, `arguments` and `assess` call runs the new
`harness/protocol_child.py` as `python -I -B` under a 120 second deadline, in
its own process group or Windows Job, and returns one strict JSON envelope.
The child compiles only protocol bytes that match the frozen SHA-256, and a
protocol `SystemExit` keeps its exit code (argparse usage errors still exit 2).
An exception the protocol raises is recorded as before R3, under its own type and
message: an `assess` failure settles as `"<Type>: <message>"`, an `arguments`
failure keeps its type and message in `coordinator-error.json` and settles through
the existing coordinator path, and a `validate_arguments` failure prints its own
message at `session start`. Transport, identity and result failures are recorded
as `TaskProtocolError` with their kind.
After every operation the last 4 KiB of the child's stderr, protocol prints
included, is copied to the caller's stderr; the scratch directory is then
deleted. The 16 MiB envelope limit and the stderr tail apply when Harness reads
the files, and the deadline bounds the run.
`task_protocol.protocol()` is removed; callers use `task_protocol.run()`. A
task `optimization.py` must take and return JSON values.
- `moe-forward-npu`: `processes.py` is folded into `benchmark.py`
(`MARKER`, `cleanup` and `_members` unchanged) and leaves `TASK_FILES`.
- `qwen3-tts-npu`: `evaluation.py` and the CLI are one `benchmark.py`; the
prepared bundle no longer has `evaluation.py`. A bundle prepared before this
change fails its frozen digests and must be prepared again. Retained results
are not rewritten.
- The MoE grader and Qwen21 evaluator keep loading the frozen candidate by path in
their own process, which timing requires. The gate admits
`importlib.util.module_from_spec` only in task bundle programs, files directly
inside a `benchmark/tasks/<bundle>/` directory without `__init__.py`, and names no
task, so a new bundle needs no gate change. Package code may not import a bundle
program. Name-based lookups (`__import__`, `import_module`, `find_spec` in any
spelling) still fail there. `first_wave_report().runtime_execution`
lists each bundle program that loads by path, because static isolation does not
cover what the loaded code does at run time.
- The optimization session's runtime identity adds `harness/protocol_child.py`,
and `harness/task_protocol.py` changed, so a session started before this
change does not resume after it.

- Six Harness modules stop importing Infra (Refs #412, R2). `agent_sessions`,
`candidates`, `optimization`, `optimization_session`, `project` and
`task_protocol` take `digest`, `sha256` and `file_lock` from the new
Expand Down
2 changes: 1 addition & 1 deletion docs/contracts/current-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,7 @@ Current status after R1 (#412):
- The seven root leaves are deleted, so `compatibility_pairs` in [`../legacy-knowledge-manifest.json`](../legacy-knowledge-manifest.json) is empty. The nine pair records stay in its `revision_history`, unchanged, as the record of this decision.
- The former ninth pair, `infra.inspection` → `inferrsi.resources.repository_packs`, is a live import that no root deletion removes. It is now a permanent exact rule outside the allowlist: that one module binds the one name `repository_pack` with one plain module-level import. The leaf's bytes, SHA-256, allowed imports, shape and import-free parent initializer stay locked, and no `resources.` or `contracts.` prefix is allowed.
- Root `arguments.py` → the same leaf, for `REPO_TYPES`, was a separate temporary entry with its own blocker. A1 (#408) removed both with the root file, and no Apps importer of the leaf replaced them.
- `overall isolation_complete` is still false. A1 (#408) cleared that temporary entry; twelve `harness` → `inferrsi.infra` hits in six Harness modules (R2) and five declared runtime resource loaders (R3) remain. The package root holds eight tracked files; the final two-file root waits for A2 (#408) and M1 (#452).
- `overall isolation_complete` is true since R3 (#412). A1 (#408) cleared that temporary entry, R2 cleared the twelve `harness` → `inferrsi.infra` hits in six Harness modules and R3 cleared the five runtime resource loaders (a child-process protocol, two folded sibling modules and a permanent two-evaluator rule). The package root holds the two tracked files `__init__.py` and `__main__.py`.
- The permanent rule and that manifest revision require formal architecture review. The implementer does not self-approve them, and passing tests are not that review.

[`../module-layout.md`](../module-layout.md) has the exact rules and their negative controls.
Expand Down
8 changes: 4 additions & 4 deletions docs/engineering-benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,10 +47,10 @@ inferrsi benchmarks run qwen21-denoise-npu --help
inferrsi benchmarks run qwen3-tts-npu --help
```

MoE owns `benchmark.py`, `grader.py` and its torchrun cleanup helper. The application
adapter injects the generic Harness executor into its entry. Python callers
provide the `execute_trial` argument explicitly. TTS owns `benchmark.py` and
`evaluation.py`, alongside its inference and service implementation. The image
MoE owns `benchmark.py` (which holds its torchrun cleanup helper) and `grader.py`. The
application adapter injects the generic Harness executor into its entry. Python callers
provide the `execute_trial` argument explicitly. TTS owns one `benchmark.py` with its
CLI and evaluation, alongside its inference and service implementation. The image
and TTS entries also run directly by file path.

## Directory ownership
Expand Down
93 changes: 55 additions & 38 deletions docs/module-layout.md
Original file line number Diff line number Diff line change
Expand Up @@ -175,29 +175,52 @@ directories have no `__init__.py`. They import only the standard library and
external engines, so the Knowledge closure rule holds for them unchanged. K02
did not change the gate's scan scope or any closure rule. A dynamic import (`__import__`, `importlib.import_module`, or
`module_from_spec`) inside a checked closure is an undeclared edge and fails
closed, except for the exact resource execution declarations below.
closed, except for by-path loading in task bundle programs, described below.
Other dynamic forms (`getattr`, `exec`, `eval`, `runpy`, `pkgutil`)
are out of scope for this package. `tools/` and `tests/` are separate
registered consumers; this gate does not scan them.

The task integration declares these existing resource execution boundaries in
`tests/_gate_policy_support.py`, independently of the Knowledge compatibility
pairs. Each permits exactly one `importlib.util.module_from_spec` call, with the
listed local spec variable, in the named function or module scope. Other dynamic
calls, additional calls, adjacent modules and static cross-core imports remain
checked. These declarations do not make arbitrary candidate code isolated.

| Relative module | Scope | Local spec |
| --- | --- | --- |
| `harness.task_protocol` | `protocol` | `spec` |
| `benchmark.tasks.moe-forward-npu.benchmark` | module | `_process_spec` |
| `benchmark.tasks.moe-forward-npu.grader` | `load` | `spec` |
| `benchmark.tasks.qwen21-denoise-npu.benchmark` | `load` | `spec` |
| `benchmark.tasks.qwen3-tts-npu.benchmark` | module | `_evaluation_spec` |

The report adds a `runtime resource loaders` blocker when these loaders are
present and does not claim either complete isolation or new-core closure
isolation. R1 changed none of these five declarations; closing them is R3 (#412).
#412 R3 retired the five temporary resource-loader declarations and their
`runtime resource loaders` blocker. Three of the five were removed:

- `harness.task_protocol` no longer loads task code. Each protocol operation
(`validate_arguments`, `arguments`, `assess`) runs `harness/protocol_child.py`
as `python -I -B` in its own process group or Windows Job under a deadline, and
answers with one strict JSON envelope on a private copy of stdout. The child
imports only the standard library and compiles the exact protocol bytes it has
checked against the frozen SHA-256. After every operation the last 4 KiB of the
child's stderr is copied to the caller's stderr; size limits apply when Harness
reads the child's output files, and the deadline bounds the run. This is an
interpreter and process boundary, not a filesystem or network sandbox.
- `moe-forward-npu` folded its sibling `processes.py` into `benchmark.py`, and
`qwen3-tts-npu` folded its sibling `evaluation.py` into `benchmark.py`. The
bundles no longer load a sibling module by path, and a bundle prepared before
R3 is refused by its frozen digests and must be prepared again.

An evaluator that times submitted code must call it in its own process, so it loads
the frozen run file by path. The gate decides this by file location, not by task:

- A **task bundle program** is a file directly inside `benchmark/tasks/<bundle>/`
where that directory has no `__init__.py`. Python could still import such a file
through a namespace package, so the gate rejects every import of a bundle program
from package code; Harness and Apps run it as its own process.
- In a bundle program, `importlib.util.module_from_spec` calls are admitted. Every
name-based lookup still fails there: `__import__`, `importlib.import_module`,
`module_from_spec` reached by another spelling, and `find_spec` in any spelling,
including an alias. Static imports of Knowledge, Harness or Apps fail as in any
Benchmark module.
- Everywhere else, including a deeper file inside a bundle, a bundle directory with an
`__init__.py`, `knowledge/data/**` scripts, Harness and Apps, the same call is a
dynamic import and fails.

No table lists tasks or functions, so a new bundle needs no gate change. Today the
MoE grader and the Qwen21 evaluator load this way. Their callers verify the file
identities (`identity.json` in both tasks), and each load returns a fresh module that
is not registered in `sys.modules`. The gate cannot see the path a bundle program
loads, so the rule is a location-based static policy about where by-path loading is
allowed, not a process or sandbox guarantee for candidate code. `first_wave_report().runtime_execution` lists every
bundle program that loads by path, so the static `isolation_complete` result never
reads as runtime isolation.

R1 (#412) deleted the three root forwarders of the old run chain and removed
their `LEGACY_RUN_FORWARDERS` registration from `tests/_gate_policy_support.py`.
Expand All @@ -215,23 +238,18 @@ With the seven Knowledge shells and root `inspection`, their names are now
No `legacy run forwarders` blocker exists any more.
[Retired root modules](#retired-root-modules) gives each replacement.

A clean result means no new cross-core import of those names. It does not mean
the modules are isolated. Isolation stays false while any one of these holds:
`knowledge/`, `benchmark/`, or `apps/` is absent or has no scannable
module; the compatibility allowlist
is non-empty;
`harness/**` still imports Model or Infra,
directly or through another module; a declared runtime resource loader is
present; or the scan reports a violation. The three packages may appear in stages.
A clean graph for the new cores is reported separately as "new core business
closure isolated" and never sets overall `isolation_complete` to true.

On the tree after R2 (#412), `first_wave_report()` has no violation and one
blocker, so `isolation_complete` is false and R2 does not claim otherwise:

| Blocker | Cause | Closed by |
| --- | --- | --- |
| `runtime resource loaders: 5` | The five declared loaders in the table above | R3 (#412) |
A clean result means no new cross-core import of those names. Isolation stays false
(`isolation_complete`, a static import-closure result) while any one of
these holds: `knowledge/`, `benchmark/`, or `apps/` is absent or has no scannable
module; the compatibility allowlist is non-empty; `harness/**` still imports Model
or Infra, directly or through another module; or the scan reports a violation. A
clean graph for the new cores is reported separately as "new core business closure
isolated". Neither flag says anything about what frozen task or candidate code does
once it runs: that is `runtime_execution`.

On the tree after R3 (#412), `first_wave_report()` has no violation and no
blocker, `new_core_closure_isolated` and `isolation_complete` are both true, and
`runtime_execution` lists the two bundle programs that load files by path.

R1 cleared the `allowlist` and `legacy run forwarders` blockers, and A1 cleared
the `temporary leaf importers` blocker that R1 left for root `arguments.py`. R2
Expand All @@ -241,8 +259,7 @@ cleared `harness reaches model/infra` (12 hits): `agent_sessions`, `candidates`,
and `file_lock` from `harness/artifact_primitives.py` and `atomic_write_json` from
`harness/atomic_json.py`, whose function bodies equal those in `infra/artifacts.py`.
`infra/artifacts.py` stays for its other users. No Harness module reaches
`inferrsi.infra` or `inferrsi.model`. The loaders also keep
`new_core_closure_isolated` false.
`inferrsi.infra` or `inferrsi.model`. R3 cleared `runtime resource loaders: 5`.

`knowledge-model-policy` stays the only required status. Until a separate
process decision makes this import gate required, every migration PR attaches
Expand Down
5 changes: 2 additions & 3 deletions docs/tts-benchmark.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,7 @@ SHA256;启动器阻止导入 vLLM、vLLM-Omni、SGLang 和历史独立引擎

| 文件 | 做什么 | 优化时的作用 |
| --- | --- | --- |
| `benchmark.py` | 解析 prepare/serve/collect/assess 参数 | 题包自身的可执行入口 |
| `evaluation.py` | 冻结判据、采集音频和计时、离线判定 | 评测规则和结果复核 |
| `benchmark.py` | 解析 prepare/serve/collect/assess 参数;冻结判据、采集音频和计时、离线判定 | 题包自身的可执行入口,也是评测规则和结果复核 |
| `baseline.py` | 校验模型资产、启用引擎导入隔离、启动服务 | 固定参考入口,记录代码与权重身份 |
| `inference.py` | 文本和指令编码 → Talker/code predictor → codec 解码 → PCM | 推理调度、缓存、算子、同步和解码优化的入口 |
| `service.py` | HTTP/SSE、请求校验、有界队列、单模型线程 | 批处理、排队和增量音频交付的入口 |
Expand All @@ -39,7 +38,7 @@ SHA256;启动器阻止导入 vLLM、vLLM-Omni、SGLang 和历史独立引擎
→ 独立 ASR/音色评分 → assess 核对 → 反馈下一轮 → 独立 final
```

这里的 prepare、serve、collect、assess 由题包内 benchmark.py 和 evaluation.py 实现。RSI 的实验调用方
这里的 prepare、serve、collect、assess 由题包内 benchmark.py 实现(#412 R3 把原来的 evaluation.py 并入它;此前准备好的 prepared 目录因冻结摘要不符会被拒绝,需要重新 prepare)。RSI 的实验调用方
负责触发它们;描述任务的 Markdown 不会自动启动评测。task、运行入口和判据对应
#362 的 Benchmark 职责,当前沿 scenario 入口提供,后续公共契约按 S1 对接。

Expand Down
83 changes: 75 additions & 8 deletions src/inferrsi/benchmark/tasks/moe-forward-npu/benchmark.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@

import argparse
import hashlib
import importlib.util
import json
import math
import os
import signal
import statistics
import sys
import time
Expand All @@ -19,16 +19,83 @@
from inferrsi.infra.artifacts import atomic_write_json
from inferrsi.model.protocols.metrics import read_metric

_process_spec = importlib.util.spec_from_file_location(
"moe_task_processes", Path(__file__).resolve().with_name("processes.py")
)
_processes = importlib.util.module_from_spec(_process_spec)
_process_spec.loader.exec_module(_processes)
MARKER, cleanup = _processes.MARKER, _processes.cleanup
# Worker cleanup for this MoE invocation, including detached torchrun sessions.
# torchrun workers may create their own sessions, outside the executor's process
# group. Only the launcher receives a fresh marker; it cannot reach the invoking
# shell or allocator. This is lifecycle supervision for reviewed code, not a sandbox.
# The marker must be generated in this driver after it starts, as evaluate() does.
MARKER = "INFERRSI_MOE_RUN_ID"


def _members(token: str, proc: Path = Path("/proc"), *, errors=None) -> list[dict]:
marker = (MARKER + "=" + token).encode()
found = []
# A fresh marker can only be inherited by processes started after this
# driver. Older same-user services may have unreadable environments; they
# cannot own this invocation. Keep same-tick processes in scope because
# Linux start times have tick granularity, not a strict ordering within it.
driver_fields = (proc / "self" / "stat").read_text().rsplit(")", 1)[1].split()
driver_start = int(driver_fields[19])
# A missing procfs must raise, never be interpreted as successful cleanup.
for entry in proc.iterdir():
if not entry.name.isdigit() or int(entry.name) == os.getpid():
continue
try:
if entry.stat().st_uid != os.getuid():
continue
fields = (entry / "stat").read_text().rsplit(")", 1)[1].split()
if fields[0] == "Z":
continue
start_ticks = int(fields[19])
if start_ticks < driver_start:
continue
if marker in (entry / "environ").read_bytes().split(b"\0"):
found.append({"pid": int(entry.name), "start_ticks": start_ticks})
except (FileNotFoundError, ProcessLookupError):
continue
except (OSError, ValueError, IndexError) as exc:
if errors is None:
raise
# Preserve confirmed workers even when another process is unreadable.
# The caller must keep the resulting cleanup verdict inconclusive.
errors.add(str(exc))
return found


def cleanup(token: str, *, grace_seconds: float = 5.0) -> dict:
"""Clean a marker minted by this driver; preserve PID-reuse protection."""
killed = set()
errors = set()
deadline = time.monotonic() + grace_seconds
try:
while True:
members = _members(token, errors=errors)
if not members:
return {"verified": not errors, "signalled_pids": sorted(killed),
"remaining": [], **({"error": "; ".join(sorted(errors))} if errors else {})}
for item in members:
try:
fields = Path(f"/proc/{item['pid']}/stat").read_text().rsplit(")", 1)[1].split()
if int(fields[19]) == item["start_ticks"]:
os.kill(item["pid"], signal.SIGKILL)
killed.add(item["pid"])
except (FileNotFoundError, ProcessLookupError):
pass
except (OSError, ValueError, IndexError) as exc:
errors.add(str(exc))
if time.monotonic() >= deadline:
remaining = _members(token, errors=errors)
return {"verified": False, "signalled_pids": sorted(killed),
"remaining": remaining,
**({"error": "; ".join(sorted(errors))} if errors else {})}
time.sleep(0.05)
except (OSError, ValueError, IndexError) as exc:
return {"verified": False, "signalled_pids": sorted(killed), "error": str(exc)}


TASK_FILES = (
"task.json", "TASK.md", "baseline.py", "oracle.py", "workload.py", "LICENSE",
"benchmark.py", "grader.py", "processes.py",
"benchmark.py", "grader.py",
)


Expand Down
Loading
Loading