Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions docs/runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,9 @@ if __name__ == "__main__":

The default exit mode remains `ouro`, with the original cumulative hazard policy. For random lookahead, sigmoid is a direct score for exiting after one more loop, not a hazard/CDF. `SamplingParams.exit_threshold` compares this score directly. Threshold 1 disables adaptive exit; `max_loops` always caps execution. Adaptive lookahead cannot exit before round 2, although an explicitly configured hard maximum of 1 is honored. All prompt tokens still run the model's full depth.

Ordinary asynchronous execution requires `ouro_delayed`, `random_lookahead` or `trace`; speculative execution has the separate constraints below. It automatically enables reusable static inputs to avoid blocking device metadata transfers. CPU supports the same scheduling state machine for tests, with immediately completed operations; it does not simulate GPU parallel execution. `multi_stream` is an independent CUDA execution feature: `False` uses one stream for core, boundary and copies; `True` separates those streams, with or without async scheduling.
Fixed-loop models accept the request-level `SamplingParams.target_loops` field (offline CLI: `--request-target-loops`). An omitted target defaults to `total_ut_steps`; an explicit target must be between one and the model depth. If `max_loops` is provided, it must match the fixed target. For compatibility with the default, `min_loops=2` is accepted, as is `min_loops` equal to the target; the fixed target determines the exit depth. Targets below the model depth are rejected until per-request prompt prefill is enabled; they are never silently run at full prompt depth. Dynamic-exit models do not accept this field and retain their existing loop-bound and exit-threshold behavior.

For dynamic-exit models, ordinary asynchronous execution requires `ouro_delayed`, `random_lookahead` or `trace`; fixed-loop models do not need an exit signal. Speculative execution has the separate constraints below. Asynchronous execution automatically enables reusable static inputs to avoid blocking device metadata transfers. CPU supports the same scheduling state machine for tests, with immediately completed operations; it does not simulate GPU parallel execution. `multi_stream` is an independent CUDA execution feature: `False` uses one stream for core, boundary and copies; `True` separates those streams, with or without async scheduling.

CLI flags are shared by offline and serving entrypoints:

Expand Down Expand Up @@ -306,7 +308,7 @@ of that capacity can safely be promised to new work.

## Replay and validation commands

Trace mode expects `depths_by_request`, with one depth per output: full prefill depth first, then each decode input's exit depth. Admission validates coverage and loop bounds. Request IDs must match the trace. The public engine replay fixes depth decisions; token generation still follows the configured sampler.
Trace mode expects `depths_by_request`, with one depth per output: full prefill depth first, then each decode input's exit depth. Trace decisions take precedence over a valid request-level `target_loops` value; the trace controls per-output depths. Trace depths remain bounded by model depth and any supplied `max_loops`. Admission validates coverage and loop bounds. Request IDs must match the trace. The public engine replay fixes depth decisions; token generation still follows the configured sampler.

```python
ExitConfig(mode="trace", depths_by_request={"A": [4, 2, 4]})
Expand Down
3 changes: 2 additions & 1 deletion docs/serving.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,7 @@ curl http://127.0.0.1:8000/v1/completions \
| `temperature`, `top_p`, `top_k` | 0, 1, -1; passed to `SamplingParams` |
| `seed`, `ignore_eos` | 0, false; request-local RNG and EOS policy |
| `min_loops`, `max_loops`, `exit_threshold` | 2, model depth, 1; Ouro extensions |
| `target_loops` | null; fixed-loop models default to model depth; trace replay takes precedence when enabled; otherwise shallower targets are rejected pending prefill support, and `max_loops` must match when supplied |
| `stream` | false; true enables SSE |
| `stream_options.include_usage` | false; valid with streaming |
| `n`, `best_of`, `echo` | only 1, 1, false |
Expand All @@ -81,7 +82,7 @@ curl http://127.0.0.1:8000/v1/completions \
Unsupported fields/behaviors, invalid values, empty tokenized prompts, impossible
KV reservations and context overflow return 400. An unknown model returns 404.
Missing or invalid model identifiers return 400. Omit fields to use defaults;
explicit null is accepted only for `max_tokens`, `max_loops`, `stream_options` and the four
explicit null is accepted only for `max_tokens`, `max_loops`, `target_loops`, `stream_options` and the four
null-only fields above. Structured errors use the same value for `error.type`
and `error.code`; admission messages include diagnostic context/KV limits.
The HTTP default uses the model context length minus the number of encoded
Expand Down
26 changes: 26 additions & 0 deletions tests/test_auto_model.py
Original file line number Diff line number Diff line change
Expand Up @@ -406,6 +406,32 @@ def test_cli_requires_source_for_real_model(monkeypatch):
assert error.value.code == 2


def test_cli_request_and_speculative_targets_are_distinct(monkeypatch):
import vllm_rlt.entrypoints.cli as cli

client = Mock()
client.engine.model_config.total_ut_steps = 4
client.generate.return_value = []
llm = Mock(return_value=client)
monkeypatch.setattr(cli, "LLM", llm)
monkeypatch.setattr(
"sys.argv",
[
"vllm-rlt",
"--toy",
"--speculative-tokens",
"1",
"--target-loops",
"4",
"--request-target-loops",
"3",
],
)
cli.main()
assert llm.call_args.kwargs["speculative_config"].target_loops == 4
assert client.generate.call_args.args[1].target_loops == 3


@pytest.mark.parametrize("answer", ["yes", "no"])
def test_missing_tokenizer_prompts_before_download(monkeypatch, answer):
from vllm_rlt.models import load_tokenizer
Expand Down
209 changes: 201 additions & 8 deletions tests/test_dory.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
import torch

from tests.reference.dory import dense_dory_reference
from vllm_rlt import CacheConfig, ExecutionConfig, SamplingParams, SchedulerConfig
from vllm_rlt import LLM, CacheConfig, ExecutionConfig, ExitConfig, SamplingParams, SchedulerConfig
from vllm_rlt.attention import BackendCapabilities
from vllm_rlt.attention.backend import AttentionBackend
from vllm_rlt.core.kv_cache_manager import KVCacheManager
Expand Down Expand Up @@ -589,26 +589,219 @@ def test_engine_serves_fixed_loops_and_matches_greedy_reference(model):
assert result.token_ids == tokens[len(prompt) :]


def test_engine_accepts_request_target_loops(model):
target_loops = model.config.total_ut_steps
engine = LLMEngine(
model,
cache_config=CacheConfig(num_blocks=256, block_size=2),
scheduler_config=SchedulerConfig(max_num_seqs=1, max_num_batched_tokens=4),
)
params = SamplingParams(max_tokens=2, target_loops=target_loops, ignore_eos=True)
engine.add_request("request", [4, 7, 3], params)
request = engine.scheduler.requests["request"]
assert request.target_loops == target_loops
assert request.loop_limit(model.config.total_ut_steps) == target_loops
finished = {}
while engine.has_unfinished_requests():
for output in engine.step():
if output.finished:
finished[output.request_id] = output
assert finished["request"].exit_depths == [target_loops, target_loops]


@pytest.mark.parametrize("mode", ["ouro", "ouro_delayed", "random_lookahead"])
@pytest.mark.parametrize("asynchronous", [False, True])
def test_fixed_request_ignores_adaptive_exit_threshold(model, mode, asynchronous):
engine = LLMEngine(
model,
cache_config=CacheConfig(num_blocks=256, block_size=2),
scheduler_config=SchedulerConfig(max_num_seqs=1, max_num_batched_tokens=4),
exit_config=ExitConfig(mode=mode),
execution_config=ExecutionConfig(async_scheduling=asynchronous),
)
engine.add_request(
"request",
[4, 7, 3],
SamplingParams(max_tokens=2, target_loops=2, exit_threshold=0, ignore_eos=True),
)
finished = {}
while engine.has_unfinished_requests():
for output in engine.step():
if output.finished:
finished[output.request_id] = output
assert finished["request"].exit_depths == [2, 2]


@pytest.mark.parametrize("explicit", [False, True])
def test_single_loop_model_accepts_default_and_explicit_target(explicit):
torch.manual_seed(42)
model = DoryForCausalLM(tiny_dory_config(n_recurrent_loops=1)).eval()
engine = LLMEngine(
model,
cache_config=CacheConfig(num_blocks=64, block_size=2),
scheduler_config=SchedulerConfig(max_num_seqs=1, max_num_batched_tokens=4),
)
params = SamplingParams(
max_tokens=2,
target_loops=1 if explicit else None,
max_loops=1 if explicit else None,
ignore_eos=True,
)
prompt = [4, 7, 3]
engine.add_request("request", prompt, params)
assert engine.scheduler.requests["request"].target_loops == 1
finished = {}
while engine.has_unfinished_requests():
for output in engine.step():
if output.finished:
finished[output.request_id] = output
result = finished["request"]
assert result.exit_depths == [1, 1]
tokens = list(prompt)
for _ in range(2):
logits = dense_dory_reference(model, torch.tensor(tokens))["logits"]
tokens.append(int(logits[-1].argmax()))
assert result.token_ids == tokens[len(prompt) :]


def test_default_target_uses_model_depth_above_minimum_default():
model = DoryForCausalLM(tiny_dory_config(n_recurrent_loops=4)).eval()
engine = LLMEngine(
model,
cache_config=CacheConfig(num_blocks=64, block_size=2),
scheduler_config=SchedulerConfig(max_num_seqs=1, max_num_batched_tokens=4),
)
engine.add_request("request", [4, 7, 3], SamplingParams(max_tokens=2, ignore_eos=True))
assert engine.scheduler.requests["request"].target_loops == 4
finished = {}
while engine.has_unfinished_requests():
for output in engine.step():
if output.finished:
finished[output.request_id] = output
assert finished["request"].exit_depths == [4, 4]


def test_llm_passes_request_target_across_stage_process(model):
with LLM(model, cache_config=CacheConfig(num_blocks=64, block_size=2)) as llm:
result = llm.generate(
[[4, 7, 3]], SamplingParams(max_tokens=2, target_loops=2, ignore_eos=True)
)
assert result[0].exit_depths == [2, 2]
with pytest.raises(ValueError, match="per-request prefill support"):
llm.generate([[4, 7, 3]], SamplingParams(max_tokens=1, target_loops=1))


def test_engine_accepts_legacy_fixed_loop_bounds(model):
engine = LLMEngine(
model,
cache_config=CacheConfig(num_blocks=256, block_size=2),
scheduler_config=SchedulerConfig(max_num_seqs=1, max_num_batched_tokens=4),
)
params = SamplingParams(max_tokens=1, min_loops=2, max_loops=2, ignore_eos=True)
engine.add_request("request", [4, 7, 3], params)
assert engine.scheduler.requests["request"].target_loops == model.config.total_ut_steps
engine.abort_request("request")


def test_engine_defaults_request_target_loops_to_model_depth(model):
engine = LLMEngine(
model,
cache_config=CacheConfig(num_blocks=256, block_size=2),
scheduler_config=SchedulerConfig(max_num_seqs=1, max_num_batched_tokens=4),
)
engine.add_request("request", [4, 7, 3], SamplingParams(max_tokens=1, ignore_eos=True))
assert engine.scheduler.requests["request"].target_loops == model.config.total_ut_steps
engine.abort_request("request")


def test_engine_rejects_invalid_request_target_loops(model):
with pytest.raises(ValueError, match="target_loops"):
LLMEngine(
model,
cache_config=CacheConfig(num_blocks=256, block_size=2),
scheduler_config=SchedulerConfig(max_num_seqs=1, max_num_batched_tokens=4),
).add_request(
"request",
[4, 7, 3],
SamplingParams(max_tokens=1, target_loops=model.config.total_ut_steps + 1),
)
with pytest.raises(ValueError, match="per-request prefill support"):
LLMEngine(
model,
cache_config=CacheConfig(num_blocks=256, block_size=2),
scheduler_config=SchedulerConfig(max_num_seqs=1, max_num_batched_tokens=4),
).add_request("request", [4, 7, 3], SamplingParams(max_tokens=1, target_loops=1))
with pytest.raises(ValueError, match="max_loops"):
LLMEngine(
model,
cache_config=CacheConfig(num_blocks=256, block_size=2),
scheduler_config=SchedulerConfig(max_num_seqs=1, max_num_batched_tokens=4),
).add_request(
"request",
[4, 7, 3],
SamplingParams(max_tokens=1, target_loops=1, min_loops=1, max_loops=2),
)
with pytest.raises(ValueError, match="min_loops"):
LLMEngine(
model,
cache_config=CacheConfig(num_blocks=256, block_size=2),
scheduler_config=SchedulerConfig(max_num_seqs=1, max_num_batched_tokens=4),
).add_request(
"request",
[4, 7, 3],
SamplingParams(
max_tokens=1,
target_loops=1,
min_loops=3,
ignore_eos=True,
),
)


@pytest.mark.parametrize("asynchronous", [False, True])
def test_trace_overrides_fixed_request_target(model, asynchronous):
engine = LLMEngine(
model,
cache_config=CacheConfig(num_blocks=64, block_size=2),
scheduler_config=SchedulerConfig(max_num_seqs=1, max_num_batched_tokens=4),
exit_config=ExitConfig(mode="trace", depths_by_request={"request": [2, 1]}),
execution_config=ExecutionConfig(async_scheduling=asynchronous),
)
engine.add_request(
"request",
[4, 7, 3],
SamplingParams(max_tokens=2, target_loops=1, ignore_eos=True),
)
request = engine.scheduler.requests["request"]
assert request.target_loops is None
finished = {}
while engine.has_unfinished_requests():
for output in engine.step():
if output.finished:
finished[output.request_id] = output
assert finished["request"].exit_depths == [2, 1]


def test_engine_rejects_unqualified_dory_configuration(model):
with pytest.raises(ValueError, match="max_loops"):
LLMEngine(
model,
cache_config=CacheConfig(num_blocks=256, block_size=2),
scheduler_config=SchedulerConfig(max_num_seqs=2, max_num_batched_tokens=4),
).add_request("request", [4, 7, 3], SamplingParams(max_tokens=1, min_loops=1, max_loops=1))
with pytest.raises(ValueError, match="min_loops and max_loops"):
# A fabricated gate of -1e4 becomes 0.0 after sigmoid, so an unchecked
# exit_threshold=0 would let a shallow request "exit" after min_loops.
).add_request(
"request",
[4, 7, 3],
SamplingParams(max_tokens=1, min_loops=1, max_loops=1),
)
with pytest.raises(ValueError, match="min_loops"):
LLMEngine(
model,
cache_config=CacheConfig(num_blocks=256, block_size=2),
scheduler_config=SchedulerConfig(max_num_seqs=2, max_num_batched_tokens=4),
).add_request(
"request",
[4, 7, 3],
SamplingParams(
max_tokens=1, min_loops=1, max_loops=2, exit_threshold=0.0, ignore_eos=True
),
SamplingParams(max_tokens=1, min_loops=1, max_loops=2, target_loops=2, ignore_eos=True),
)
with pytest.raises(ValueError, match="last_exited"):
LLMEngine(
Expand Down
33 changes: 32 additions & 1 deletion tests/test_engine.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
import torch

from tests.helpers import tiny_ouro_config
from vllm_rlt import LLM, CacheConfig, SamplingParams, SchedulerConfig
from vllm_rlt import LLM, CacheConfig, ExitConfig, SamplingParams, SchedulerConfig
from vllm_rlt.engine.llm_engine import LLMEngine
from vllm_rlt.models import OuroForCausalLM
from vllm_rlt.request import FinishReason, Request, RequestOutput, RequestPhase
Expand Down Expand Up @@ -183,6 +183,36 @@ def test_same_id_after_finish_starts_from_a_fresh_generator():
assert second["r"].exit_depths == first["r"].exit_depths


@pytest.mark.parametrize("depth", [1, 2, 4])
def test_request_fixed_target_is_read_only(depth):
request = Request("request", [2, 4], SamplingParams(), _target_loops=depth)
assert request.target_loops == request.loop_limit(4) == depth
request.loops_done = 1
assert request.target_loops == depth
with pytest.raises(AttributeError):
request.target_loops = 3


def test_target_loops_are_rejected_for_dynamic_models():
with pytest.raises(ValueError, match="fixed-loop models"):
LLMEngine(tiny_model(), cache_config=CacheConfig(64, 2)).add_request(
"request", [2, 4], SamplingParams(max_tokens=1, target_loops=1)
)


def test_trace_takes_precedence_over_dynamic_target():
engine = LLMEngine(
tiny_model(),
cache_config=CacheConfig(64, 2),
exit_config=ExitConfig("trace", depths_by_request={"request": [4, 2]}),
)
engine.add_request(
"request", [2, 4], SamplingParams(max_tokens=2, target_loops=1, ignore_eos=True)
)
outputs, _ = drain(engine)
assert outputs["request"].exit_depths == [4, 2]


def test_abort_drops_the_rng_slot_so_id_reuse_starts_over():
params = SamplingParams(max_tokens=4, temperature=0.8, seed=23, ignore_eos=True)
engine = LLMEngine(tiny_model(), cache_config=CacheConfig(64, 2))
Expand Down Expand Up @@ -318,6 +348,7 @@ def test_integer_limits_rejected_before_scheduling(invalid):
(SamplingParams, "max_tokens"),
(SamplingParams, "min_loops"),
(SamplingParams, "max_loops"),
(SamplingParams, "target_loops"),
(SchedulerConfig, "max_num_seqs"),
(SchedulerConfig, "max_num_batched_tokens"),
(CacheConfig, "num_blocks"),
Expand Down
2 changes: 2 additions & 0 deletions tests/test_pd.py
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,8 @@ def ready_client(model, *, stage_config, **options):
engine.add_request(
"bad_depth", [1], SamplingParams(max_loops=config.total_ut_steps + 1)
)
with pytest.raises(ValueError, match="PD does not support per-request target_loops"):
engine.add_request("target", [1], SamplingParams(target_loops=config.total_ut_steps))
with pytest.raises(ValueError, match="exceeds context capacity"):
engine.add_request(
"bad_context", [1], SamplingParams(max_tokens=config.max_position_embeddings + 1)
Expand Down
Loading
Loading