Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,12 @@ Portable operator guidance and candidate procedures live in [src/inferrsi/knowle

## Optimization requests

For requests to use benchmarks to verify RSI knowledge efficiency, start with
`inferrsi benchmarks compare --help` and follow its returned `next_action`.
The default knowledge comparison runs Qwen denoising and MoE forward, each
on/off in independent sessions. Configuration, native handoff, trace and report
details are in [benchmark comparison](docs/benchmark-comparison.md).

Ordinary project optimization stays in the current Agent conversation. Follow
the [shared optimization workflow](docs/optimization-workflow-unification.md):
prepare explicit workload, quality/measurement scripts, metric, gates, declared
Expand Down
1 change: 1 addition & 0 deletions docs/apps-module-migrations.json
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
"docs/module-layout.md",
"docs/operations.md",
"src/inferrsi/apps/api.py",
"src/inferrsi/apps/benchmark_compare.py",
"src/inferrsi/apps/delivery/__init__.py",
"src/inferrsi/apps/handoff/bridge.py",
"src/inferrsi/apps/handoff/candidate_provider.py",
Expand Down
164 changes: 164 additions & 0 deletions docs/benchmark-comparison.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,164 @@
# Compare RSI optimization efficiency

Use the self-describing Apps entry when asked to use benchmarks to determine
whether RSI knowledge improves optimization efficiency. No extra skill is needed.

```bash
inferrsi benchmarks compare --help
inferrsi benchmarks compare
```

Knowledge is the default focus. Its default task set is
`qwen21-denoise-npu` and `moe-forward-npu`, each with knowledge **on/off**.
One repeat means four independent Agent optimizations, not four measurements
of a prewritten candidate. Repeated `--task` replaces the default task set.
Each execution Agent chooses changes from feedback, completes the exact round
budget, and independently retests its selected code with `session final`.

## Resolve and freeze configuration

CLI values override `.inferrsi/benchmarks.json` in the current workspace, then
built-ins. `--config` selects another non-secret JSON file. The resolved values
and their sources are frozen in `experiment.json`; each job references its
digest. Resume with the returned `--output`, never by preparing over that path.

Built-in budgets are eight rounds, one repeat, 60 campaign minutes, 1,200 seconds
per evaluation/final reserve, and 90 minutes per Agent job. These are bounded
run settings, not universal performance gates. Select a prepared execution
environment before dispatch. An unconfigured entry returns
`configuration_required` with an actionable request; it does not fabricate an
NPU host. Read task requirements with `benchmarks show`. A private profile can be:

```json
{
"schema_version": 1,
"environment_id": "prepared-ascend-pool",
"evaluation_prefix": "npu run --gpus {devices} --timeout 18m --wait 1m --",
"execution_instructions": "Use the existing authorized target. Return each complete campaign tree to its declared local workspace.",
"rounds": 8,
"repeats": 1,
"policy": {
"metrics": ["speedup", "wall_seconds"],
"tolerances": {"speedup": 0.0, "wall_seconds": 0.0, "tokens": 0.0},
"minimum_pairs": 1
}
}
```

`{devices}` comes from each task: Qwen needs one NPU, MoE needs two. Default task
arguments are Qwen `--pairs 2 --samples 3 --warmups 1` and MoE
`--pairs 2 --timeout-seconds 900`. Override one with
`--evaluation-args 'TASK=ARGUMENTS'`. Keep allocator wait/run/cleanup within the
outer evaluation timeout. Optional `model` and `effort` are requests, not proof
of actual host settings. Credentials never belong in this configuration.

## Drive the host handoff

The command manages experiment records. The initiating Agent calls its actual
host tools for fresh-session creation and observation, following `next_action`.
This is the same native handoff as [Agent sessions](native-agent-sessions.md).
The CLI neither starts a hidden Agent CLI nor invents a conversation identity.

1. Prepare: `benchmarks compare --config <profile.json> --output <new-directory>`.
2. Read `next_action`; call `compare --action dispatch --output <directory>
--trial <id>`. Only `may_create: true` permits one native creation call.
3. Send the exact returned prompt with the declared workspace, and supported
requested model/effort. Bind the confirmed native ID using `--action bind`,
`--host`, `--session-id`, optional `--host-id`, and `--receipt <host.json>`.
Each trial needs a distinct session ID within the host, including across
repeats. Report verification excludes trials that share a binding.
4. Observe that same session. Retain actual host observations with
`--action record --observation <observation.json>`. Uncertain creation stays
`dispatching`; inspecting the original call takes precedence over relaunch.
5. After termination and artifact return, call `--action collect --trial <id>`.
Keep failed/incomplete results. By default, trials run in declared order,
with on/off order alternating between repeats. With sufficient devices,
pass `--parallel` on dispatch for each trial you want to start while others
are active. Keep each allocation separate and note concurrent execution
when interpreting campaign wall time.
6. Generate a progress or final comparison with `--action report`. Read
`--action status` to resume. Reporting never executes missing rounds/finals.

All non-prepare actions require `--output`. Dispatch/bind/record/collect also
require `--trial`. Jobs are created just before dispatch so waiting for earlier
trials does not consume their host deadlines. Native deadline cancellation
still belongs to the initiating Agent and its supported host tools.

Native observation JSON uses the existing `agents record` format, plus:

```json
{
"host": "<HOST>",
"host_id": null,
"session_id": "<NATIVE_SESSION_ID>",
"state": "completed",
"native_receipt": {"replace_with": "actual native tool response"},
"benchmark_observation": {
"model": "<ACTUALLY_OBSERVED_MODEL>",
"effort": "<ACTUALLY_OBSERVED_EFFORT>",
"environment_id": "prepared-ascend-pool",
"environment_fingerprint": "<DIGEST_OF_NON_PRIVATE_RUNTIME_AND_DEVICE_PROFILE>",
"knowledge_policy_violation": false
}
}
```

These remain caller-exported observations, not authenticated host attestations.
Retain their underlying receipts. Include actual `usage_scope: session_total`
and `usage` counters when available; cached input is a subset of input. Final
host usage is never added to campaign round subtotals. Missing usage stays null.
Do not replace missing final counters with an earlier partial observation.

## Isolation and evidence

Every trial has a separate workspace and frozen runtime. Shared code and task
bytes match; on receives the pinned package corpus, off omits `knowledge/data`
and disables managed session retrieval. Both prompts exclude other skill
libraries, external knowledge tools, web research and prior trial solutions.
Public task/CLI instructions and ordinary target APIs remain available to both.
This is runtime-corpus isolation plus an instruction boundary; it is **not an OS
filesystem/network sandbox**. Report observed violations and bound conclusions
to this managed-knowledge treatment. Do not claim all latent or external
knowledge was removed. Remote trials return the exact full campaign tree;
arbitrary summaries cannot substitute for retained evidence.

Collection verifies the frozen task/protocol, budget, knowledge condition,
runtime identity, raw evidence and independent final candidate binding. Paired
actual model, effort and environment must match. Remote interpreter paths may
differ. Compute the environment fingerprint from the observed Python
implementation and version, framework and runtime versions, device model and
allocator profile. `python` selects the executable in the launch command.
Report generation verifies the campaign again to detect changes after collection.

Each round can supply `--strategy <description>` and repeated
`--knowledge-ref <delivered-context-path>` in addition to `--hypothesis`.
Harness accepts citations only to contexts already delivered in that campaign.
The trace labels these as Agent declarations; delivery does not prove use.
Unrecorded citations remain unknown, rather than treating every previous search
as knowledge used by every round.

## Read the report

Apps writes a new JSON/Markdown/offline-HTML report snapshot under `reports/`.
Start with the conclusion and per-task comparison, then inspect complete traces
(strategy, hypothesis, knowledge citations, round/best speedup, correctness and
selection), independent finals and evidence gaps. JSON digest links the two
readable documents to the same snapshot. Original campaign evidence is retained.

Benchmark rules compare paired on/off benefit factors: on/off speedup, off/on
wall time or tokens. The default efficiency basis is final optimization speedup
and actual campaign wall time; token usage is displayed separately. Add tokens
to policy.metrics only with comparable whole-session observations. Tolerances
are frozen relative neutral bands, not statistical significance tests.

Conclusions distinguish observed improvement, no gain, regression, tradeoff and
insufficient evidence. Missing/invalid declared pairs stay in denominators and
make the overall comparison insufficient. Statistics are per-task medians of
paired log ratios; unrelated task latencies are never pooled. Results are
descriptive within the declared tasks, budgets and repeats, never a general
knowledge-efficacy or model/service E2E claim. Set campaign-specific minimum
repeats and neutral tolerances before search when stronger evidence is needed.

Apps owns configuration, composition and presentation; Benchmark owns task and
comparison rules; Harness owns optimization execution and retained facts;
Knowledge owns the delivered material. No new core-to-core imports are added.
61 changes: 40 additions & 21 deletions docs/moe-forward-benchmark.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,37 +88,56 @@ CPU 输入/oracle 准备、计时前 barrier、验收、形状预热不在 forwa

## 在 RSI 优化循环里如何使用

```text
用户目标与题目约束 → Agent 改候选 → 本入口验算并测量 → Agent 看证据再修改
↓
调用方保留每轮结果、选最佳版本、独立复测
MoE 题包提供 `optimization.py`,可以使用与 Qwen 相同的 Harness 完整迭代流程。
它复用单候选入口的同一份双 rank 原始证据规则,不根据 rank 0 汇总或 Agent 自报分数选优。

执行 Agent 在已经准备好的环境中启动:

```bash
inferrsi benchmarks session start moe-forward-npu \
--session ./outputs/moe-session \
--rounds <rounds> --max-minutes <budget-minutes> \
--evaluation-timeout-seconds 1500 --final-reserve-seconds 1500 \
--condition <condition-label> --knowledge <on-or-off> \
--evaluation-prefix 'npu run --gpus 2 --timeout 20m --wait 1m --' \
--evaluation-args '--pairs 2 --timeout-seconds 900'
```

调用一次评测一个候选。`inferrsi benchmarks list` 和
`inferrsi benchmarks show moe-forward-npu` 可以读取题包信息,执行使用本页的 MoE 入口。
调用方负责启动 Agent、推进轮次、统计模型 tokens 和选择搜索策略,
并将每次 `result.json` 作为下一轮反馈。
知识库 ON/OFF 或不同 Agent 的差异由调用方控制,测量协议保持一致。
profiling 可作为 Agent 诊断工具,评测入口不要求先 profiling。
轮数与总预算由本次实验声明;总预算需要容纳 final 预留。这里的采样和超时是可覆盖的
命令示例。每轮和 final 使用同一份冻结的配对数与完整 case 集合,评测器内部超时、
分配器运行/等待超时和外层评测超时应逐层留出启动与清理余量。

Agent 阅读返回的 `frozen/TASK.md`,修改 `work/candidate.py`;每次修改后提交假设:

```bash
inferrsi benchmarks session round --session ./outputs/moe-session \
--candidate ./outputs/moe-session/work/candidate.py \
--hypothesis '<本轮优化策略与理由>'

# 完成声明的轮次后,独立复测已留存的最佳源码
inferrsi benchmarks session final --session ./outputs/moe-session
```

可给新 Agent 的简短任务指令:
知识库开启时使用 `benchmarks session knowledge` 检索并留存实际交付内容;关闭时拒绝
受管检索。外部来源和直接文件读取的隔离由对照实验的执行环境与 Agent 访问约定负责。
Profiling 由 Agent 按瓶颈选择,不是每轮必经步骤。

> 阅读 `src/inferrsi/benchmark/tasks/moe-forward-npu/TASK.md`,
> 优化 `./implementation.py` 的完整双 NPU MoE forward。遵守输入只读、专家所有权、
> 正确性和计时边界。每次候选使用新的输出目录,按已约定的配对数、超时和实验预算,
> 通过 `inferrsi benchmarks run moe-forward-npu` 取得结果;保留失败,不以自报数字替代结果。
> 最终交付最佳候选、逐轮证据和独立复测记录。
Harness 留存每轮源码、假设、实际评测、历史最佳选择和独立 final。迭代 assessor 核对
候选、全部题目/协议文件、grader、launcher 与采样请求的哈希,重算两个 rank 的完整
调用与计时样本,供标准结果交付和优化 trace 使用。正常结束的候选错误保留为失败轮;
内部或外部执行中断、缺少 rank 证据、清理未确认以及基线失败均阻断后续搜索。
恢复通过 `session recover` 回收原有尝试,不重新执行同一候选。

例如要做 8 轮再复测,由调用方为 R1…R8 和 final 分配不同目录并保存候选快照。
轮数是该次实验的预算,不能把它当成通用默认值。优化曲线按每轮有效分数及历史最佳分数
绘制;失败轮保留空分数并消耗预算。累计 tokens 需从对应 Agent/session 的原始 usage
记录汇总;本入口的 `elapsed_seconds` 是评测墙钟,不是 Agent 搜索墙钟或模型费用。
启动新会话与回收使用通用 [Agent 交接](native-agent-sessions.md);
[共享迭代流程](optimization-benchmarks.md) 说明预算、用量导入和标准交付。
单独的 `benchmarks run moe-forward-npu` 仍只评测一个候选。完整过程的 token 需从
真实宿主 usage 导出;本题 `elapsed_seconds` 仅是评测墙钟,不是 Agent 搜索消耗。

## 验证范围与限制

历史冻结题目、baseline、oracle、采样逻辑已有双 NPU 实验;本次发布保留了这些数学和
计时语义,加入便携包装、源码冻结、双 rank 证据复核和独立 session 清理。
**新增的便携入口尚未在 NPU 上重新跑通**,CPU 测试和 wheel 资源检查只验证接线、
**新增的迭代协议尚未在 NPU 上重新跑通**,CPU 测试和 wheel 资源检查只验证接线、
失败处理及证据完整性,不能当作新入口的 NPU 验收或性能结论。

当前权重更新用例验证原位更新后的结果,但两次访问之间还运行其他参数用例;
Expand Down
1 change: 1 addition & 0 deletions docs/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ Effect classes used below:

| Intent | Entry | Effect class | Boundary |
| --- | --- | --- | --- |
| Verify whether RSI knowledge improves optimization efficiency | CLI `benchmarks compare`; [comparison workflow](benchmark-comparison.md) | Local experiment records / explicit native handoff / report-generating | Apps freezes two-task on/off plans, returns host actions, verifies retained finals and reports Benchmark comparisons; no hidden Agent launch |
| Complete a task optimization in the current Agent | CLI `benchmarks session start/knowledge/round/status/final/recover` | Experiment-executing / local records | Frozen budget, actual candidates and independent task outcomes. The current Agent drives edits. Real usage is imported explicitly. Recovery collects an existing execution |
| Optimize an ordinary project in the current Agent | CLI `campaign start/knowledge/round/status/final/recover` | Experiment-executing / local records | Harness freezes explicit caller-provided checks, manages multi-file candidates, executes and assesses retained results. This path does not depend on Benchmark or infer acceptance from the goal. See [shared workflow](optimization-workflow-unification.md) |
| Prepare and record a fresh native Agent | CLI `agents prepare/dispatch/bind/record/status` | Local records / explicit host handoff | The initiating Agent calls native host tools. Records prevent repeated dispatch; exported host observations do not establish task acceptance. See [native sessions](native-agent-sessions.md) |
Expand Down
8 changes: 6 additions & 2 deletions docs/optimization-benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,14 @@ only observes and collects its results. Benchmark supplies task resources and
task-specific rules. Harness freezes those inputs, runs the checks, assesses
retained outputs and manages the lifecycle; it does not invent project acceptance.

The first task with this session protocol is `qwen21-denoise-npu`. MoE and TTS
currently expose their single-candidate task entries. New session tasks provide
The iterative tasks are `qwen21-denoise-npu` and `moe-forward-npu`; TTS
currently exposes its single-candidate task entry. New session tasks provide
their own `optimization.py` alongside the task definition and evaluator.

For paired knowledge on/off experiments over both tasks, use the Apps
[`benchmarks compare` workflow](benchmark-comparison.md). It reuses this session
lifecycle and collects complete optimization traces and independent finals.

## Start the complete workflow

In the execution conversation, run in the prepared NPU environment. The example declares eight optimization
Expand Down
Loading
Loading