Skip to content

[feat] deploy: add hardware-aware vLLM-Omni setup - #259

Open
WEIXILIN0307 wants to merge 17 commits into
ThinkFlowLab:mainfrom
WEIXILIN0307:feat/vllm-omni-deployment
Open

WEIXILIN0307 wants to merge 17 commits into
ThinkFlowLab:mainfrom
WEIXILIN0307:feat/vllm-omni-deployment

Conversation

@WEIXILIN0307

@WEIXILIN0307 WEIXILIN0307 commented Nov 10, 2026 •

Copy link
Copy Markdown

Summary

Local model setup requires a separate deployment workflow. This PR brings hardware-aware setup and verification into the installed OpenDirector CLI, helping users and Agents connect a local model and produce project assets without a source checkout.

  • Inspect the assigned GPU and host resources before installing into a separate model environment.
  • Ship deployment/verification commands, pinned model profiles and service lifecycle tooling in the Python package.
  • Verify image, speech and transcription output, then use existing OpenDirector operations to deliver editable project assets.
  • Provide reusable English/Chinese guides and a repository-hosted deployment Skill for external Agents.

Implementation

Start with the CLI adapter, then review plan, start/stop and verify.

Capability Approach Key entry
Host inspection and resource checks Model target: Linux x86_64/Python 3.12. Read-only GPU/driver, VRAM, RAM and disk checks; require an explicit GPU within CUDA visibility. deploy.py: inspect / plan
Isolated installation Separate omni-env and caches; pin top-level runtime/compiler packages, probe CUDA/FlashInfer and record the environment. deploy.py: environment / install
Model download Pinned snapshots via Hugging Face SDK or the existing resumable range downloader; verify sizes and available SHA256 values. deploy.py: download / register_snapshot; profiles
Service lifecycle Per-profile lock, loopback binding, PID/start-time identity and health/model-ID checks. Startup timeouts preserve the original process; stop sends group SIGTERM. deploy.py: start / status / stop
Output verification Readiness by default; --infer submits once, saves a receipt, decodes PNG pixels, probes audio metadata or compares ASR text. No automatic inference retry. verify.py
Installed CLI and resources opendirector vllm-omni deploy / verify, with --help; package shared tool sources, profiles and downloader without adding GPU runtimes to application dependencies. vllm_cli.py; setup.py; MANIFEST.in
Application asset handoff Use existing generation/import/adoption commands. Preserve the asset name in canvas add-material so imported transcripts remain editable. cli.py: add_material; test_cli.py

Review process ownership, receipt provenance and package boundaries first. Project delivery uses existing application operations; deployment helpers never write project files. The material-title fix was discovered during ASR acceptance and could be reviewed separately.

Validation

Scope: fe50e27. Audit reruns are marked below; other results come from reviewed local implementation logs/receipts, not uploaded CI artifacts. No new full-suite or GPU run was performed; model checks do not establish production readiness.

Check Result and evidence scope
Latest complete Python run 2,308 passed, 100 skipped, 7 failed in 444.51 s. Log reviewed; full suite remains unpassed.
CLI/deployment regression 78 passed, audit rerun: test_vllm_omni_cli.py, test_vllm_omni_deployment.py, test_cli.py; covers discovery, preflight, lifecycle boundaries, unknown submissions and output validation.
Failed-case rechecks 6 passed, 1 failed in the audit with private temporary directories. CI-profile cases passed; test_native_runtime.py::test_rejects_removed_runtime_and_invalid_selection_before_acceptance still returned 503 instead of 409. Earlier isolated passes do not resolve this; a probe reported invalid OpenCode skill-catalog JSON.
Frontend and production build 832 frontend tests passed; production build passed during wheel-from-sdist build. Logs reviewed.
Static checks Audit rerun: 4 import contracts kept, 3,033 i18n keys / 0 problems, browser locale and manifest checks passed.
Wheel / sdist Inspected 0.1.4 sdist/Linux x86_64 wheel; tool/profile bytes match this head. Audit reran verify_install.py outside the checkout: help, profiles, downloader plan, application/web startup and project persistence passed; uv pip check passed.
Editable installation Implementation-phase smoke passed before the latest main merge; not repeated at this head.
Real GPU workflow Historical direct/application runs below; predates CLI packaging and final main merge.
GitHub Actions Reviewer routing passed. Release checks and CodeQL are action_required, with no jobs started.

Reproduction and troubleshooting: the English / Chinese guides.

Real-model Evidence

November 9–10, 2026: Ubuntu 22.04.5, Python 3.12.7, driver 580.126.09; one RTX PRO 6000 Blackwell Server Edition (SM120, 97,887 MiB), not an Ampere A6000. Recorded software: vLLM/vLLM-Omni 0.32.0, Torch 2.13.0+cu130, CUDA 13.0. CUDA, FlashInfer and dependency checks passed.

Model / snapshot Direct verifier result OpenDirector application evidence
Z-Image-Turbo — f332072aa78be7aecdf3ee76d5c247082da564a6 Decoded 1344×768 PNG; 9 steps, CFG 0, seed 7; recorded elapsed 2.766 s. CLI/UI images registered, adopted and reopened. Visual review: harbor scene present; imperfect hull text and moored boats.
Qwen3-TTS-12Hz-0.6B-CustomVoice — 85e237c12c027371202489a0ec509ded67b5e4b5 Bilingual vivian MP3; 24 kHz mono, 8.304 s of audio; recorded elapsed 6.158 s. CLI/UI audio registered, adopted, reopened and played. A participant reported one UI sample complete, without missing words or other defects noticed.
Qwen3-ASR-0.6B — 5eb144179a02acc5e5ba31e748d22b0cf3e303b0 Native vLLM transcription of three TTS samples; each matched all 54 normalized reference characters. No language/reference hint was sent to the model. Transcript manually registered with text-asset/canvas commands, edited in the UI and retained after restart.
Composition and persistence Generated image + narration exported through the existing browser renderer: H.264/AAC, 640×360, 10.026667 s; full FFmpeg decode passed. Application restart preserved assets, jobs, adoption and editor state; saved file fingerprints matched and the video played to completion.

Verifier elapsed time ends when the response is received; decoding/hashing follows, and ASR also includes input probing. These single requests are not throughput or end-to-end deployment benchmarks.

Participant report: assisted cached setup, no help requests/retries, SSH recovery and assets retained after reopening. First success was estimated retrospectively at 5–10 minutes; exact timing, total manual actions and recovery duration were not recorded.

The existing screenshot shows the reopened project before the later application-process restart:

Reopened project with generated image, narration, timeline, export and edited ASR transcript

Limitations

  • Eligibility does not establish measured compatibility. VRAM reservations—44 GiB image, 32 GiB TTS, 16 GiB ASR—are not measured minima or workload isolation. Multi-GPU serving, laptops, Ampere SM86, Windows and Apple Silicon deployment are untested.
  • Tested topology: separate application host via loopback SSH forwards. Public/authenticated hosting, supervision, cold-install performance, sustained load and allocator peaks are unvalidated. Export required an open browser.
  • Model-ID readiness does not attest remote weights: receipt revisions come from the profile. ASR uses native vLLM; application registration is manual. ASR agreement/listening feedback applies only to these samples.
  • Open lifecycle finding (P1): a controlled local process-group probe reproduced stop returning stopped once the leader exited while a SIGTERM-ignoring descendant remained alive. stop currently checks the leader identity; complete descendant shutdown is not established.
  • Other findings (P2): external-URL PNG validation still requires root/omni-env/bin/python; incomplete state can raise unstructured exceptions; the wheel includes unused FastH3 helper modules. No code was changed by this audit.

Related Issue / Review Notes

  • Refs #107 and the claimed deployment scope: provides the initial Linux/Blackwell setup, resource/inference checks, guides and participant trial. Missing measurements and validation gaps remain; this does not close the roadmap.
  • Installed-CLI review: commands/help/profiles/downloader work from the installed wheel without a checkout. The external-Agent Skill remains repository-hosted.
  • Validation-record review: one-off record/screenshot files are absent from the current diff; reusable guides remain. Evidence is here, with the screenshot embedded from a fixed historical commit rather than a new attachment. Both threads remain unresolved; no maintainer approval is implied.

Comment thread tools/vllm_omni/deploy.py Outdated
return dict(status='stopped')


def main():

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we expose deployment and verification through the installed opendirector CLI, with --help and packaged profiles, so users and Agents can use them without a source checkout?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Deployment and verification are now available through opendirector vllm-omni deploy and verify, with --help, packaged profiles, and the range downloader included. I verified the wheel installation outside the source checkout as well.

This record separates resource eligibility, actual inference, project delivery
and creative review. 本记录分别记录资源预检、真实推理、工程交付与创作品质审看,不能互相代替。

## Environment / 环境

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we move this validation record and docs/development/assets/vllm-omni-validation.png to the PR description as text and attachments, keeping reusable instructions in the guides?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, thanks! I moved the validation results and screenshot to the PR description and removed the one-off files from the repository. The reusable English and Chinese guides remain in the repo.

@github-actions github-actions Bot added the area:docs Documentation / 文档 label Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:docs Documentation / 文档 area:install Install, startup and environment support area:provider Model providers and generation protocols type:feat New capability or behaviour

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants