Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions docs/en/configuration/experiments.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,51 @@ tasks:
graph_node_classification: {}
```

## Explicit continuation to a finite timeout

The default auto-resume policy stops after at most 100 resumes. A long run
whose Agent repeatedly exits successfully can reach that ceiling before its
time budget. To use the timeout as the ceiling for successful exits, opt in:

```bash
sforge run --task <task-id> --agent codex --timeout 64800 --resume-until-timeout
```

The Python `run_agent` API accepts `resume_until_timeout=True`. The mode
requires an explicit positive integer `timeout` argument or `agent_timeout`
configuration and an Agent resume command. It conflicts with
`--disable-auto-resume`. The same container, registration and Agent resume
command are retained; submission limits, cooldown and scoring are unchanged.

One monotonic deadline starts after initial command preparation at the first
solver dispatch. Initial preparation and final collection remain separate
costs. Later command preparation and time between segments consume the same
deadline; resumes never renew it. Stop and remaining time are checked again
after preparation, immediately before dispatch. Cancellation, any nonzero process exit, an Agent-owned
terminal blocker, a timed-out segment, or a segment shorter than one second
can still end work early. This mode does **not** guarantee a minimum duration.
Keep any time deficit and failure evidence instead of treating an early exit
as a completed budget. Backend timeout drain can add collection cost after
solver work stops.

Stock Agents inherit no additional classifier for successful process output.
A quota message with exit code zero requires an explicit Agent
`resume_block_reason` implementation to block continuation; this flag does not
infer that condition from arbitrary text.

The opt-in run writes `resume-policy.json`; the CLI's `final_result.json` and
Python result include the same `resume_policy`. It records the original
deadline, actual resumes, process time, solver-phase elapsed time, full
invocation elapsed time, stop reason and unused seconds. No policy field or
file is added to default runs. To disable the mode, omit the flag or pass
`False` in Python.

This is in-process continuation, not checkpoint restoration. An existing
`started_at` or policy receipt rejects a new opted-in invocation in that run
directory. Preserve the old attempt; a cold restart needs a new run identity
and remains a separate experiment. Existing frozen runs are not upgraded or
relabelled by enabling this flag on a later run.

## Fields

### `env`
Expand Down
32 changes: 32 additions & 0 deletions docs/zh/configuration/experiments.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,38 @@ tasks:
graph_node_classification: {}
```

## 显式按有限超时继续原会话

默认自动续跑最多100次。长任务若反复正常退出,可能在时间预算尚未耗尽时触及上限。
需要以有限超时约束正常退出后的续跑时,显式启用:

```bash
sforge run --task <task-id> --agent codex --timeout 64800 --resume-until-timeout
```

Python `run_agent` 接受 `resume_until_timeout=True`。该模式要求显式传入正整数
`timeout` 或配置 `agent_timeout`,并且Agent有原生续跑命令;与
`--disable-auto-resume` 冲突。容器、registration和Agent续跑命令保持同一条链,
提交次数限制、冷却与评分不变。

第一次命令准备完成后,在solver dispatch边界确定唯一的monotonic截止时间。
首次准备与最后收集分别计成本,后续命令准备和段间开销消耗同一截止时间;续跑不会
重新计时。准备完成、实际dispatch前再次检查Stop和剩余时间。Stop、非零退出、Agent明确的终态阻塞、
单段超时及不足一秒的短段仍会提前结束,因此这**不是最低时长保证**。提前结束时须
保留欠时和失败证据。后端的超时收尾可能在solver停止后继续消耗收集时间。

默认Agent未新增对成功进程输出的quota解析;退出码为零的quota信息如需阻止续跑,
须由Agent的 `resume_block_reason` 明确提供,此flag不会从任意文本推断该条件。

显式模式写入 `resume-policy.json`;CLI的 `final_result.json` 和Python结果包含
相同的 `resume_policy`,记录原截止时间、实际续跑数、进程时长、solver阶段时长、
本次完整调用时长、停止原因和未用秒数。默认运行不增加该字段或文件。省略flag或
在Python传入 `False` 即关闭。

该模式只在同一次调用内续跑,不负责checkpoint恢复。已有 `started_at` 或policy
receipt的目录拒绝再次以该模式启动。保留原attempt;冷启动需要新的run身份,独立
计为另一实验。后继启用新flag不会升级或重标已有冻结运行。

## 字段说明

### `env`
Expand Down
7 changes: 6 additions & 1 deletion sforge/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -246,6 +246,7 @@ def _run_single_task(
disable_stop_hook=disable_stop_hook,
disable_auto_eval=disable_auto_eval,
disable_auto_resume=disable_auto_resume,
resume_until_timeout=getattr(args, "resume_until_timeout", False),
internet=internet,
verbose=verbose,
shutdown_event=shutdown_event,
Expand Down Expand Up @@ -886,8 +887,12 @@ def main():
dest="disable_auto_eval", help="Disable background auto-evaluation daemon")
p_run.add_argument("--disable-stop-hook", action="store_true", default=False,
dest="disable_stop_hook", help="Disable the agent stop hook (agent can exit normally)")
p_run.add_argument("--disable-auto-resume", action="store_true", default=False,
resume_group = p_run.add_mutually_exclusive_group()
resume_group.add_argument("--disable-auto-resume", action="store_true", default=False,
dest="disable_auto_resume", help="Disable auto-resume on abnormal agent exit")
resume_group.add_argument("--resume-until-timeout", action="store_true", default=False,
help="Resume successful exits in the same session until the explicit timeout; "
"failure, stop and short-segment guards still apply (default: max 100 resumes)")
p_run.add_argument("--max-submissions", type=int, default=None, dest="max_submissions",
help="Maximum number of agent submissions per run (default: unlimited)")
p_run.add_argument("--submission-cooldown", type=int, default=None, dest="submission_cooldown",
Expand Down
10 changes: 10 additions & 0 deletions sforge/harness/agent/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@

from sforge.harness.agent.bundle import ensure_bundle, inject_bundle
from sforge.harness.backend import ContainerBackend, ContainerHandle
from sforge.harness.backend.base import StreamingExecResult
from sforge.harness.config import SForgeConfig


Expand Down Expand Up @@ -91,6 +92,15 @@ def augment_env(self, env: dict[str, str], model: str | None) -> None:
mirrors, extra_env) have already been applied.
"""

def resume_block_reason(self, result: StreamingExecResult) -> str | None:
"""An Agent-owned terminal blocker for explicit deadline continuation.

Inspect only the current invocation. The runner also stops on any
nonzero exit, cancellation, timeout or short segment in that mode.
Returning None does not authorize a new session or a renewed budget.
"""
return None

def format_run_cmd(
self,
prompt_path: str,
Expand Down
Loading