Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ This directory is the entry point for the repository's shared documentation. The
| [Python installation and packaging](development/packaging.md) | Installation, platform wheel builds, and isolated verification | Packaging, resource paths, or build process changes |
| [Business CLI](guides/cli.md) | Querying, editing, and generating through a terminal or Agent Shell | Commands, authorization, or receipt semantics change |
| [DeepSeek Harness plugin](../plugins/deepseek-harness/README.md) | Installing the DSH bundle and editing an OpenDirector project from DSH | Plugin setup, tools, or creation workflows change |
| [Video model capabilities](guides/video-model-capabilities.md) | Declaring output specifications supported by a service | Configuration fields or capability integration changes |
| [Video model capabilities](guides/video-model-capabilities.md) ([中文](guides/video-model-capabilities.zh-CN.md)) | Declaring output specifications supported by a service | Configuration fields or capability integration changes |
| [OpenAI Images](guides/openai-images.md) | Wiring gpt-image as an optional image service, request shape and failures | Integration fields, capability scope, or the OpenAI interface change |
| [Kling video](guides/kling-video.md) ([中文](guides/kling-video.zh-CN.md)) | Optional Kling 4.0 and Flash setup, model-specific inputs, and recovery limits | Kling inputs, model declarations, or recovery behavior change |
| [BigModel video](guides/bigmodel-video.md) ([中文](guides/bigmodel-video.zh-CN.md)) | CogVideoX-3 text, single-image and first/last-frame generation, task recovery and adoption | BigModel protocol, inputs, output sizes or recovery behavior change |
Expand Down
4 changes: 4 additions & 0 deletions docs/architecture/agent-runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,4 +69,8 @@ Default 的模型网关在 `agent-runs/<turn-id>.model.jsonl` 记录实际转发

工具允许列表与操作系统沙箱提供不同约束。业务工具独立校验工程范围和输入;原生文件访问权限不授予直接改写工程存储的业务权限。

Windows 上的 Codex 启动显式设置 `windows.sandbox="elevated"`,避免隔离启动忽略用户配置后,`workspace-write` 降为只读并拦截 Shell。保留沙箱约束,不绕过审批或沙箱。PowerShell 使用 `Get-Command opendirector` 发现 CLI,POSIX Shell 使用 `command -v opendirector`。

On Windows, Codex explicitly sets `windows.sandbox="elevated"` so ignoring user configuration does not downgrade `workspace-write` to read-only and block Shell execution. Sandbox restrictions remain enabled. Discover the CLI with `Get-Command opendirector` in PowerShell or `command -v opendirector` in POSIX shells.

CLI 通过环境变量中的本轮本地 Socket 地址连接业务网关,不使用连接令牌或凭据文件。网关继续校验工程范围、轮次生命周期及业务输入。
2 changes: 1 addition & 1 deletion docs/guides/user-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ example vLLM's 4096 default) rejects every request with HTTP 400.

For vLLM-Omni video connections, enter fractional seconds directly, for example `6.583333`. The CLI accepts `opendirector video create --duration 6.583333` together with the other required options. Read `opendirector video create --help` and `opendirector model list` first; `inputRules.fractionalDuration` identifies support, and each output's duration list defines the allowed range. H3 can align output to its supported frame count, so inspect the actual result duration before assembling segments.

For detailed video settings, see [Video model capability configuration (Chinese)](video-model-capabilities.md). When no explicit output specification is provided, use **Auto** or **Service default duration**.
For detailed video settings, see [Video model capability configuration](video-model-capabilities.md). When no explicit output specification is provided, use **Auto** or **Service default duration**.

Automatic region selection in images and video matting require a model with segmentation capabilities. Image generation and editing still require the corresponding generation or editing model.

Expand Down
2 changes: 1 addition & 1 deletion docs/guides/user-guide.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Kimi 对话助手有两种计费方式,各自使用独立的 Key 和服务地
若服务端的 `--max-model-len` 更小(例如 vLLM 的 4096 默认值),每次请求都会返回
HTTP 400。

视频模型的详细配置见 [视频模型能力配置](video-model-capabilities.md)。未提供明确输出规格时,使用“自动”或“服务默认时长”。
视频模型的详细配置见 [视频模型能力配置](video-model-capabilities.zh-CN.md)。未提供明确输出规格时,使用“自动”或“服务默认时长”。

自动识别图片选区或视频抠像需要具备分割能力的模型;生成图片、编辑图片仍需选择对应的生成或编辑模型。

Expand Down
68 changes: 35 additions & 33 deletions docs/guides/video-model-capabilities.md

Large diffs are not rendered by default.

82 changes: 82 additions & 0 deletions docs/guides/video-model-capabilities.zh-CN.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# 视频模型能力配置

模型接入信息与输出能力分别保存于本机后端。能力是声明数据,不包含脚本,也不会修改模型权重、推理步数或服务器显存。当前实现独立于其他产品的代码。

生成历史会分别显示请求时长和成片实测时长。例如,FastH3 请求 4 秒时,为保留尾帧锚点可能生成 24 fps 的 107 帧,成片时长显示为 4.458 秒。

## 界面操作

1. 打开“模型接入”,添加或编辑视频模型,填写地址、Model ID 和鉴权信息。
2. 点击默认折叠的“视频适配方式”展开高级配置。对 `/videos` 接口选择适配方式。FastH3 使用八步和多素材参考,每次生成使用新的随机种子并随任务保存,恢复任务时沿用;FastH3 首尾帧(`fasth3_dual`)在同一模型上提供两种生成方式:参考模式与 FastH3 相同,首尾帧模式最多使用首帧、尾帧两张锚点图片,按 `fl2va`、`res_multistep` 采样、20 步提交,帧数落在 17n+5 网格上,尾帧锚点保留原生网格最后一帧、不按所选时长裁剪;首尾帧模式必须同时提供首帧和尾帧,不接受视频或音频参考,也不与多素材参考同时使用。原生 H3 支持首帧、素材参考及接续;通用适配仅发送文本和单张首帧,不携带 H3 专属的采样参数。模型使用自定义名称时,应明确选择适配方式。
3. 已知模型可点击“载入适配预设”。未声明能力的模型只显示“自动”和“服务默认时长”。
4. 开启“自定义输出规格”,每档填写分辨率、支持时长和可选横屏像素宽高。时长支持 `4-10,15`,即 4 至 10 秒以及 15 秒。不同档位可填写不同的时长集合。
5. 设置默认分辨率、默认时长和像素对齐步长,保存。是否需要填写像素尺寸取决于提交参数:发送明确宽高的适配需要配置尺寸;发送分辨率档位、由服务端确定宽高的适配可以留空。已知 H3 档位也可使用适配器内置的尺寸。H3 使用 32 像素对齐,现有适配仍限定 4–15 秒。

切换模型后使用新模型的默认分辨率;兼容的已选时长保留,否则采用新模型的默认时长。改变时长后,界面仅显示该时长支持的分辨率;不适用的原选项回到默认值或自动。后端在接受任务前再次校验,不会将不支持的组合悄悄改成其他时长。

参考素材决定画幅,显式分辨率决定输出尺寸;接续保留上一段的尺寸,仅使用自动。已有供应商的模式限制继续生效,例如 Seedance 图片参考模式的分辨率限制。自定义能力不能改变上传协议、开放未实现的参考模式,或突破原生适配的限制。

关闭“自定义输出规格”并保存可恢复适配预设。编辑时密钥留空会保留已保存的密钥,能力信息中不包含密钥。已有配置读取时保留此前适配方式,不会因为模型别名而自动切换请求格式;新增未知模型则采用通用方式。

## 读取服务声明

“读取服务声明”只请求配置服务的 `GET /models`(基础地址通常已含 `/v1`),不创建生成任务,不保存配置。用户确认并保存后才应用。它要求模型列表中对应 **精确 Model ID** 的条目包含 `videoCapabilities`。

这是 OpenDirector 的可选扩展约定,并非 OpenAI 或 vLLM 标准保证的字段。普通 `/models` 列表只有模型名称时,不会据此推断分辨率或时长,仍需手动填写。无需为了使用本功能修改推理服务器;也可完全使用本机配置。

声明示例(仅演示格式,需替换成服务真实支持的参数):

```json
{
"data": [
{
"id": "custom-video",
"videoCapabilities": {
"outputs": [
{ "resolution": "540P", "durations": [3, 5], "width": 960, "height": 540 },
{ "resolution": "720P", "durations": [5, 7], "width": 1280, "height": 720 }
],
"defaultResolution": "720P",
"defaultDuration": 7,
"dimensionStep": 2
}
}
]
}
```

每份声明最多 20 档;时长为 1–60 秒内的整数,具体可用范围取决于适配和服务。宽高须同时填写或留空;横屏宽不小于高,边长 64–8192,且满足对齐步长。默认分辨率和时长必须构成有效组合,分辨率名称不能重复。

读取失败、字段无效或服务未提供声明时保留原配置。已保存的密钥只会在基础地址完全一致时复用;读取新地址需要重新输入密钥。

## 扩展边界

已有适配协议下接入新模型、修改输出档位,只需配置。全新服务若使用不同的上传、提交、轮询或结果下载协议,仍需开发独立适配器。界面声明不会保证远端模型、账户权限或硬件能完成所配置的请求。

## MiniMax 官方视频 V2

添加视频服务时选择“MiniMax 官方视频 V2”,基础地址填写 `https://api.minimax.cn/v2`,使用 Bearer API Key。模型列表来自内置声明,载入列表不验证密钥或账户权限。连接继续由 `providerId` 管理,工程引用模型配置自身的 `id`;同名自部署模型使用独立连接与 `vllm_omni` 协议。

该适配接入 `MiniMax-H3`、`MiniMax-H3-Highspeed`、`MiniMax-H3-Lite` 和 `MiniMax-H3-Max`,按各型号限制输出档位与整数时长。首尾帧模式支持明确选择首帧、尾帧或两者;不选择帧时按文本生成。参考模式支持图片、视频和音频,包括单独音频参考。首尾帧不能与多模态参考混用。首尾帧输出比例由服务按输入决定;文本生成的自动比例解析为 `16:9`。不接受自定义像素尺寸或缓存续接。

本地素材按实际字节编码为 Base64,不上传到额外托管服务,也不自动压缩。支持 PNG、JPEG、WEBP 图片,MP4、MOV 视频和 WAV、MP3 音频;尺寸、数量、单文件大小、时长、视频帧率与编码在提交前检查。编码后的请求不得超过官方请求预算,超限明确拒绝。URL 与 `mm_file://` 输入、HEIC/HEIF 和回调通知不属于此适配范围。

任务接受后固定模型、参数、素材和参考用途。已知远端任务恢复时继续查询原任务;提交结果未知时不自动重发。结果下载为本地素材并进入既有候选与采用流程,CDN 下载不携带 API Key。停止本地执行不代表远端取消成功;本适配未接入远端取消,任务可能继续执行并计费。

协议依据:[创建任务](https://platform.minimax.cn/docs/api-reference/video-generation-v2-create)、[查询任务](https://platform.minimax.cn/docs/api-reference/video-generation-v2-query)。真实服务调用和生成质量需要使用已配置账户单独验证。

## 提供商与逐模型设置

可灵 4.0 使用独立的 `kling_video` 协议,按文生、首尾帧和图片参考选择对应官方提交路径,共用现有任务查询、恢复和结果采用流程。固定能力由后端发布。`kling-4.0` 支持 720P / 1080P、3–30 秒及首尾帧;`kling-4.0-flash` 支持 720P、3–20 秒和首帧,不支持尾帧。两个型号均支持图片参考与原生音频,不填写自定义像素尺寸。接入与输入限制见[中文指南](kling-video.zh-CN.md)和[English guide](kling-video.md)。

MiniMax 官方视频提供四个型号,新增服务时默认全部勾选,保存后才启用;编辑已有服务时保留原有勾选。标准 H3 支持 768P / 2K、4—15 秒;Highspeed 和 Lite 支持 480P / 768P、4—15 秒;Max 支持 480P / 768P、5—15 秒。各型号的输出档位与时长由后端分别校验。

添加服务时先选择提供商。提供商列表分为“自定义 API”和“vLLM-Omni 兼容”两组,各组只列出自身支持的接入类型。视频页列出 MiniMax 官方视频、可灵官方视频、火山方舟 / Agent Plan,以及 “vLLM-Omni 兼容”分组;“vLLM-Omni 兼容”分组内细分视频接口、分割接口和深度接口。选中后确定协议与接入方式,表单不再提供全局协议列表进行二次分类。高级连接设置用于填写地址等连接参数;自部署地址由用户填写。MiniMax 对话预设使用 [官方 OpenAI 兼容地址](https://platform.minimax.cn/docs/api-reference/text-openai-api),视频预设使用视频 V2 协议。

同类别、同地址与协议下存在已保存连接时,可以明确选择复用,以保留密钥和模型身份;不复用则新建独立连接。手动添加和获取列表可以交替使用,获取结果仍需勾选,已有选择不会因列表刷新消失。

图片页的 “vLLM-Omni 兼容”分组包含图片接口和分割接口;音频页包含语音合成、音乐生成和扒谱接口。各项直接确定对应协议,不再显示全局协议选择。对话、图片、音频页的“OpenAI 兼容”统一位于“自定义 API”分组首项,合并官方 OpenAI 与兼容服务接入;默认填写官方地址,地址与鉴权可编辑。

已选模型各自显示能力卡片。对话模型独立声明图片理解;视频模型独立配置适配方式和输出规格,官方 H3 与可灵展示后端发布的固定能力。编辑一个模型不会批量覆盖其他模型的能力。保存只保存配置,不执行推理测试,也不自动切换当前创作模型;对话模型可单独执行连通性测试。

首尾帧引用使用图片素材与 `role=first_frame` / `last_frame`,不填写 `subjectRole`。`subjectRole` 表示主体素材槽位,不表示视频首帧。明确指定的首尾帧若会被引用策略过滤,提交会返回 `reference_invalid`,不接受任务或退化为文生视频;未配置首尾帧的文生视频仍然允许。`fasth3_dual` 的首尾帧模式要求首帧和尾帧同时存在:缺少任一锚点或附带音频、视频参考时,提交被拒绝且不创建任务。
6 changes: 5 additions & 1 deletion packages/core/src/opendirector/agent/native.py
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,10 @@ def command(runtime: str, work: Path, socket: str, token: str, model: str | None
settings = ['features.shell_tool=true', 'features.apply_patch_freeform=false', 'web_search="disabled"',
'sandbox_workspace_write.network_access=true',
'skills.bundled.enabled=false', f'developer_instructions={json.dumps(SYSTEM)}']
if sys.platform == 'win32':
# --ignore-user-config also removes windows.sandbox. Without a
# Windows backend Codex downgrades workspace-write to read-only.
settings.append('windows.sandbox="elevated"')
args = [binary, 'exec', '--json', '--skip-git-repo-check', *codex_isolation_flags(binary), '--sandbox', 'workspace-write']
for setting in settings:
args += ['-c', setting]
Expand Down Expand Up @@ -246,7 +250,7 @@ async def stream(self, run: RunContext, gateway: ToolGateway):
args += ['--session' if runtime == 'opencode' else '--resume', session_id]
execution_context = (f'Execution directory: {work}\nNative session directory: {native_work}\n'
'Set the native shell workdir to this execution directory for file and media operations. '
'Use inherited PATH to discover installed programs with command -v. '
'Use inherited PATH to discover installed programs with Get-Command in PowerShell or command -v in POSIX shells. '
'Read source assets at their supplied absolute paths; do not move or modify them. '
'Write all intermediate files and outputs into the execution directory. '
'Use opendirector --help to discover OpenDirector commands and opendirector <group> <command> --help for usage. Use opendirector for project changes.')
Expand Down
2 changes: 1 addition & 1 deletion packages/core/src/opendirector/agent/tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ def query_guidance(failure):
SYSTEM = '''You are OpenDirector, a video creation assistant.
Respond directly when the user's request can be answered without project state or tool execution.
When the task needs project capabilities, use the harness's native shell tool to discover and run the opendirector executable on PATH.
Before using the project CLI, execute command -v opendirector through the shell, then execute opendirector --help.
Before using the project CLI, locate opendirector through the shell (Get-Command opendirector in PowerShell, command -v opendirector in POSIX shells), then execute opendirector --help.
Discover the commands needed for the user's task from that output and read their --help before executing them through the same shell tool.
After performing requested creative work, summarize the delivered result and material limitations. Keep execution IDs, revisions and receipts in tool details unless requested.
Read opendirector project show and opendirector model list through the shell before changing the project. Use exact current object revisions.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -199,6 +199,8 @@ def _parameters(self, spec, config, node, records):
seconds = intent.get('durationSeconds')
if rules['profile'] == 'fasth3-omni':
params = video_parameters(node, records, duration_seconds=seconds)
if config.get('videoAdapter') == 'fasth3_dual' and node.get('videoMode') == 'first_frame':
params['num_inference_steps'] = 20
params['seed'] = new_seed()
elif rules['profile'] == 'standard-video':
params = {}
Expand Down Expand Up @@ -374,7 +376,7 @@ def accept(before):
spec, resolved = current_generation_inputs(intermediate, spec, public.inputRules)
spec = execution_spec(spec, resolved, public.inputRules)
try:
validate_generation(spec, public, resolved)
validate_generation(spec, public, resolved, submitting=True)
if spec.kind == 'audio' and spec.parameters.creationType == 'audio':
if old != voice_spec or config != voice_config:
raise ContractError('revision_conflict', '音色验证期间生成配置或模型服务已变化,请重新提交。', spec.id)
Expand Down
2 changes: 1 addition & 1 deletion packages/core/src/opendirector/domain/creation/canvas.py
Original file line number Diff line number Diff line change
Expand Up @@ -243,7 +243,7 @@ def node_capabilities(item: CanvasItem, *, spec: GenerationSpec | None = None,
generate = False
if configure and matching and model is not None:
try:
validate_generation(spec, model, references)
validate_generation(spec, model, references, submitting=True)
generate = True
except (ValueError, ProjectError):
pass
Expand Down
Loading
Loading