Skip to content

[Dory] Establish per-request fixed-loop contracts (2A phase 1/4) - #196

Open
LyxWxj wants to merge 1 commit into
InferMatrixLab:mainfrom
LyxWxj:dory/pr-2a-fixed-loop-counts
Open

LyxWxj wants to merge 1 commit into
InferMatrixLab:mainfrom
LyxWxj:dory/pr-2a-fixed-loop-counts

Conversation

@LyxWxj

@LyxWxj LyxWxj commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Required human contributor answers

1. What outcome are you trying to achieve?

Establish the request-level fixed-loop contract for Dory as the first increment of RFC #176's staged-scheduling work. The contributor's four-phase delivery plan records the intended split. This PR prepares the contract; it does not complete heterogeneous-depth prefill or all of 2A.

2. What is the highest-leverage variable for that outcome?

The fixed recurrent loop count carried by each request. The RFC's 2A requirements call for different request depths without changing weights. The contributor specified that an omitted target uses total_ut_steps, and an explicit target cannot exceed that depth.

3. How will this work move that variable rather than create busywork?

Give admission, scheduler exit decisions and asynchronous signal handling one resolved request target, with explicit validation and regressions. This supplies the contract needed by the subsequent staged-CODA, mixed-CODA and heterogeneous atomic-prefill phases in the contributor's plan. Unsupported shallow targets fail at admission rather than silently using full-depth prompt computation.

4. Who is the customer or user, and what do they actually need?

The contributor confirmed Dory serving callers as the users: they need to specify a fixed loop count per request. This is the serving capability requested by RFC #176, PR 2A.

PR necessity and bugfix evidence

The contributor explicitly requested this first implementation PR and its submission. It adds a runtime/API contract required by the phased plan; it is not a claim of a reproduced production incident. Follow-up prefill work remains necessary before shallower request targets can execute.

Purpose

Add SamplingParams.target_loops and offline --request-target-loops, resolving fixed-model defaults and validation in StageEngine._resolve_loop_bounds. Store the effective fixed target on Request; scheduler completion and runner signal retention use that contract. Fixed requests ignore adaptive thresholds and do not require learned exit-signal readback. Trace replay takes precedence over a valid request target. Dynamic-exit models reject the new field; PD rejects it explicitly.

For example, a two-loop Dory model accepts SamplingParams(target_loops=2) or an omitted target. A target of one still raises the explicit per-request-prefill-support error. Legacy matching bounds remain accepted. Retaining only min_loops/max_loops would keep fixed-target and adaptive-bound semantics coupled; the explicit field separates their contracts while preserving the existing dynamic-exit path.

Documentation and regressions cover defaults at model depths 1/2/4, invalid values, trace priority, subprocess transport, HTTP parsing and synchronous/asynchronous execution. No staged-CODA, WCPB, mixed-depth prefill, cache reclamation or new PD support is claimed.

Test Plan

vllm-rlt Version / Commit: 0e5c15a1fed97aa333e2d0783167549c2ed8c78d; parent 8ff4ed8. The PR contains this single commit. Validation used an isolated source snapshot whose Git tree matches the commit (32dcd5bfe7b205d6db08d1e40ad5b0676fdd1288), preserving the server's existing checkout.

  • Local Ruff 0.16.8: python -m ruff check .; Python-source format check; git diff --check HEAD^ HEAD.
  • Server lab-dell03: Ruff 0.17.1, OMP_NUM_THREADS=1 python -m pytest -q, and pre-commit run --all-files using the repository's Python-file lint/format hooks and full CPU hook.
  • Reserved A100 GPU 0 with canhazgpu run --gpu-ids 0 --timeout 30m -- ...: python -m pytest -q --run-gpu tests/test_dory_gpu.py tests/test_dory_speculative.py tests/test_dory_real_checkpoint.py::test_real_checkpoint_generation_stops_on_eos_or_limit with DORY_CHECKPOINT=/home/louyuxuan/models/Dory1.0-4B-BF16.
  • Real-checkpoint HTTP smoke: one resident BF16/Triton server, readiness and model discovery, first inference, overlapping default/explicit/legacy requests, threshold-zero fixed policy, streaming text/usage/finish reason, invalid-target errors and shutdown cleanup. This is a functional smoke, not an upstream-client throughput benchmark.

Test Result

  • Ruff lint and Python formatting passed locally and on the server; all pre-commit hooks passed.
  • CPU suite: 1102 passed, 714 skipped, 148.70 s. Skips do not qualify GPU paths.
  • Dory GPU selection: 158 passed, 93.24 s, including sync/async and eager/Graph oracle agreement, cancellation/failure cleanup, speculation regressions and real-checkpoint generation.
  • Real HTTP: 5 expected HTTP 200 responses and 4 expected HTTP 400 responses; readiness in 9.15 s. Generated choices and usage agreed across default, explicit and legacy controls; streaming agreed with non-streaming output. Shutdown reported zero requests and zero used KV blocks. GPU memory returned to zero after validation.
  • Environment: Python 3.12.13, PyTorch 2.13.0+cu130, Transformers 4.55.0, Triton 3.7.1; one NVIDIA A100 80GB PCIe for GPU checks.
  • Local checkpoint identity: config SHA256 d2d086f5949598a043a00d70e2c28c45305bca7703682de3e4f990a7fd3279a4; weights SHA256 207483664dfef54c4c4438863260d3ba423bf3d6009b43fe84264b281e9fda23. A downloaded upstream snapshot revision was not independently established.
  • Raw logs, HTTP responses, reproduction script and checkpoint hashes are retained outside Git at artifacts/dory-pr-0e5c15a/ locally and /tmp/dory-pr-0e5c15a/artifacts/validation/ on the server.

Self-review applied the performance and refactoring skills: admission resolves the target; Request owns the effective value; the scheduler owns progress and exits; runner event/signal lifetime remains explicit (R02/R03/R15/R19/R21). No additional actionable code-quality issue was found. This is a correctness/contract increment with no arithmetic or kernel changes and no speedup claim. No HTTP throughput campaign, ops-only/full profiles, GSM8K-87 quality qualification or full real-weight FP32-oracle matrix was run; the targeted evidence above does not substitute for those qualifications. Standalone Ruff 0.17.1 format --check . additionally reports existing Markdown-code-block formatting in three documents; the configured Python-only formatting hook passes.

…time contracts and regressions

Signed-off-by: LyxWxj <yuxuanlou621@foxmail.com>
@LyxWxj LyxWxj changed the title [Dory] Establish per-request fixed-loop contracts (2A phase 1) [Dory] Establish per-request fixed-loop contracts (2A phase 1/4) Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant