Repository navigation
Conversation
…time contracts and regressions Signed-off-by: LyxWxj <yuxuanlou621@foxmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Required human contributor answers
1. What outcome are you trying to achieve?
Establish the request-level fixed-loop contract for Dory as the first increment of RFC #176's staged-scheduling work. The contributor's four-phase delivery plan records the intended split. This PR prepares the contract; it does not complete heterogeneous-depth prefill or all of 2A.
2. What is the highest-leverage variable for that outcome?
The fixed recurrent loop count carried by each request. The RFC's 2A requirements call for different request depths without changing weights. The contributor specified that an omitted target uses
total_ut_steps, and an explicit target cannot exceed that depth.3. How will this work move that variable rather than create busywork?
Give admission, scheduler exit decisions and asynchronous signal handling one resolved request target, with explicit validation and regressions. This supplies the contract needed by the subsequent staged-CODA, mixed-CODA and heterogeneous atomic-prefill phases in the contributor's plan. Unsupported shallow targets fail at admission rather than silently using full-depth prompt computation.
4. Who is the customer or user, and what do they actually need?
The contributor confirmed Dory serving callers as the users: they need to specify a fixed loop count per request. This is the serving capability requested by RFC #176, PR 2A.
PR necessity and bugfix evidence
The contributor explicitly requested this first implementation PR and its submission. It adds a runtime/API contract required by the phased plan; it is not a claim of a reproduced production incident. Follow-up prefill work remains necessary before shallower request targets can execute.
Purpose
Add
SamplingParams.target_loopsand offline--request-target-loops, resolving fixed-model defaults and validation inStageEngine._resolve_loop_bounds. Store the effective fixed target onRequest; scheduler completion and runner signal retention use that contract. Fixed requests ignore adaptive thresholds and do not require learned exit-signal readback. Trace replay takes precedence over a valid request target. Dynamic-exit models reject the new field; PD rejects it explicitly.For example, a two-loop Dory model accepts
SamplingParams(target_loops=2)or an omitted target. A target of one still raises the explicit per-request-prefill-support error. Legacy matching bounds remain accepted. Retaining onlymin_loops/max_loopswould keep fixed-target and adaptive-bound semantics coupled; the explicit field separates their contracts while preserving the existing dynamic-exit path.Documentation and regressions cover defaults at model depths 1/2/4, invalid values, trace priority, subprocess transport, HTTP parsing and synchronous/asynchronous execution. No staged-CODA, WCPB, mixed-depth prefill, cache reclamation or new PD support is claimed.
Test Plan
vllm-rlt Version / Commit:
0e5c15a1fed97aa333e2d0783167549c2ed8c78d; parent8ff4ed8. The PR contains this single commit. Validation used an isolated source snapshot whose Git tree matches the commit (32dcd5bfe7b205d6db08d1e40ad5b0676fdd1288), preserving the server's existing checkout.python -m ruff check .; Python-source format check;git diff --check HEAD^ HEAD.lab-dell03: Ruff 0.17.1,OMP_NUM_THREADS=1 python -m pytest -q, andpre-commit run --all-filesusing the repository's Python-file lint/format hooks and full CPU hook.canhazgpu run --gpu-ids 0 --timeout 30m -- ...:python -m pytest -q --run-gpu tests/test_dory_gpu.py tests/test_dory_speculative.py tests/test_dory_real_checkpoint.py::test_real_checkpoint_generation_stops_on_eos_or_limitwithDORY_CHECKPOINT=/home/louyuxuan/models/Dory1.0-4B-BF16.Test Result
d2d086f5949598a043a00d70e2c28c45305bca7703682de3e4f990a7fd3279a4; weights SHA256207483664dfef54c4c4438863260d3ba423bf3d6009b43fe84264b281e9fda23. A downloaded upstream snapshot revision was not independently established.artifacts/dory-pr-0e5c15a/locally and/tmp/dory-pr-0e5c15a/artifacts/validation/on the server.Self-review applied the performance and refactoring skills: admission resolves the target;
Requestowns the effective value; the scheduler owns progress and exits; runner event/signal lifetime remains explicit (R02/R03/R15/R19/R21). No additional actionable code-quality issue was found. This is a correctness/contract increment with no arithmetic or kernel changes and no speedup claim. No HTTP throughput campaign, ops-only/full profiles, GSM8K-87 quality qualification or full real-weight FP32-oracle matrix was run; the targeted evidence above does not substitute for those qualifications. Standalone Ruff 0.17.1format --check .additionally reports existing Markdown-code-block formatting in three documents; the configured Python-only formatting hook passes.