Repository navigation
[FEAT] Support random sampling in async block speculation - #187
Merged
bjf-frz merged 10 commits intoNov 17, 2026
Merged
Conversation
Add batched_probabilities in vllm_rlt/worker/speculative_sampling.py. It builds every row's temperature/top-k/top-p distribution from per-row parameters in one pass, matching Sampler.probabilities row by row, with no host read. Sampler is unchanged. Signed-off-by: ycjcx123 <ycjcx123@gmail.com>
Compare batched_probabilities with the row-wise Sampler.probabilities, covering tied top-k logits, mixed truncating and non-truncating rows and BF16 logits. Check that inputs stay unmodified and host arguments are validated. Signed-off-by: ycjcx123 <ycjcx123@gmail.com>
Add categorical, an inverse-CDF draw with a float64 CDF that always lands on a positive-weight token, and speculative_accept, which runs rejection sampling over ragged draft rows without host reads. The first draft position that is not a valid accept is the stop point: a valid rejection draws the correction from max(p - q, 0), an invalid distribution there sets the request's error flag, and later positions are ignored. Signed-off-by: ycjcx123 <ycjcx123@gmail.com>
Check categorical's support and float64 CDF edge cases. Check speculative_accept against analytic target distributions with single-draft quadrature and ragged multi-draft Monte Carlo, plus hand-checked stop-point cases. Signed-off-by: ycjcx123 <ycjcx123@gmail.com>
Plan random rows and parameters on the host, sample drafts from q, build p at verify and decide committed tokens with speculative_accept. A random round returns the accepted drafts plus the correction or bonus token, so the CPU commits what the GPU continues from; greedy rows are unchanged. Each random request draws 2K + 1 uniforms per round, so no host sync is needed. Rounds with random rows append per-request error flags; a live flag fails the round through the existing fatal cleanup before the scheduler update. The entry point still rejects random requests. Signed-off-by: ycjcx123 <ycjcx123@gmail.com>
Pin the host plan for mixed, all-greedy and zero-draft batches, the float64 bit-cast uploads and explicit-range result decoding, including stale flags in reused buffers. Check round-level results against a per-request reference and the engine's handling of stale and live flags. Signed-off-by: ycjcx123 <ycjcx123@gmail.com>
Lift the greedy-only check for async block speculation. Pipeline mode and the PD decode stage keep rejecting random sampling, and the existing rejection test now asserts the pipeline case. Signed-off-by: ycjcx123 <ycjcx123@gmail.com>
Check that accepted non-argmax drafts keep the CPU sequence, next-round GPU input and per-depth KV consistent with a serial replay. Cover in-flight, ragged, zero-draft and CUDA Graph rounds, per-request RNG ownership and replay, and fatal errors with stale-flag handling. Signed-off-by: ycjcx123 <ycjcx123@gmail.com>
Through the public API only: async block engines accept random sampling and pipeline engines reject it, random requests complete and close cleanly, the same seed replays, and greedy output is unchanged next to random requests. A new file avoids the test files changed by InferMatrixLab#153. Signed-off-by: ycjcx123 <ycjcx123@gmail.com>
Document the algorithm, result layout, RNG protocol and reproducibility scope, numerical-error semantics and support matrix. Eager and CUDA Graph execution are marked validated on one RTX 5090 (SM120) with Triton. Signed-off-by: ycjcx123 <ycjcx123@gmail.com>
ycjcx123
force-pushed
the
feat/m7-async-spec-sampling
branch
from
November 16, 2026 13:41
1c4d764 to
c9dcdff
Compare
Contributor
Author
|
Rebased onto
The full 11/15 and 11/07 prior-base validation evidence ( |
ycjcx123
marked this pull request as ready for review
November 17, 2026 03:57
Collaborator
|
Solid work, lgtm. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Required human contributor answers
1. What outcome are you trying to achieve?
Make asynchronous block speculation support random sampling (
temperature > 0,top_k,top_p) whose committed tokens still follow the target distribution.2. What is the highest-leverage variable for that outcome?
Correctness of the committed output distribution, together with not breaking the asynchronous pipeline or greedy execution.
3. How will this work move that variable rather than create busywork?
Device-side rejection sampling checked against analytically computed expectations (deterministic quadrature and Monte Carlo), CUDA tests, and end-to-end real-model comparisons, so the distribution is measured rather than argued from the construction.
4. Who is the customer or user, and what do they actually need?
Users of asynchronous speculative decoding who need temperature, top-k and top-p sampling to work there, and maintainers who need the shared result layout and RNG ownership to stay correct for greedy and synchronous decoding.
PR necessity and bugfix evidence
A PR is needed: this is a new capability, not a bugfix, so there is no observed failure to link. Async block speculation previously rejected every non-greedy request (
StageEngine._add_requestraised fortemperature != 0); this PR lifts that restriction for async block mode only and keeps the rejection for pipeline and PD decode stage.Purpose
Enable
temperature > 0,top_k, andtop_psampling in asynchronous block speculation (SpeculativeConfig(num_speculative_tokens=3, mode="block")withasync_scheduling=True), including batches containing both greedy and random requests. Previously these requests raisedValueError; they now use device-side speculative rejection sampling without host reads during submission.Three contracts govern the implementation:
K=0samples from the target.RngRegistrysupplies one fixed block of2K+1uniforms per random request per submitted round. Greedy sampling acquires no generator. Fixed-seed replay requires the same execution mode, round shapes and batch plan; equality with synchronous sampling or across eager/Graph modes is not guaranteed. Reusing the current synchronous sampler would introduce host reads, so this PR uses device-side sampling instead.Local async block execution is validated in eager and CUDA Graph modes on one RTX 5090 (SM120) with the Triton attention backend. Pipeline speculation and PD async decode remain greedy-only; FA4 is unvalidated on SM120, and async block preemption remains rejected. Usage and the algorithm are documented in
docs/async-speculative.md.Part of #43 and #66#issuecomment-6078423491: this implements and qualifies stochastic sampling for the asynchronous block mode listed as open there (
docs/runtime-history.md), building on the sampling/RNG refactor from umbrella RFC #32 (#80).Related work: #157/#164 overlap the async runner's reference lifetime, and #83/#78 overlap result/RNG interfaces. #153's 43 CUDA tests were checked on a prior-base integration snapshot (
444638b..740a097), preserved below.Issues encountered & questions for maintainers (shared with #153)
test_async_speculative_graphs.py[9],test_input_preparation.py[6],test_cuda_graph.py[12]), 27 CUDA cases hit the same paged FA4 rejection on SM120 during setup atvllm_rlt/kernels/flash_attention.py:21:ValueError: Paged FlashAttention-4 is not supported on SM120 by flash-attn-4==4.0.0b30; use --attention-backend triton until a compatible paged implementation is available.444638bfails 27/27 with the same failing node IDs and rejection message.--attention-backend tritonvalidation withflash_attn_4and multi-GPU PD (nixl) explicitly recorded as unvalidated acceptable to mark#153and this PR ready?#153or a separate test-hygiene PR)?2K + 1uniforms per round + float64 inverse-CDF): Decouples RNG consumption from acceptance outcomes so submission needs no host sync; fixed-seed replay holds within one execution path, while outputs are intentionally not token-identical to synchronous speculation or across eager/CUDA Graph modes.temperature != 0.Test Plan
vllm-rlt Version / Commit (current head):
c9dcdff0741de41f8f50bd8b9e400d2c9d370917(feat/m7-async-spec-sampling, rebased onto basec338aaa32f24b485576fafef069e00dd35627b0e, version0.2.0;git range-diffagainst prior candidate740a097is=across all 10 commits).2026-11-16,m7-rebase-logs-20261116.tar.gz): Executed on one dedicated NVIDIA GeForce RTX 5090 (32,607 MiB, SM120, driver580.76.05), Python3.12.3, PyTorch2.8.0+cu128, Triton3.4.0,--attention-backend triton, modelOuro-1.4B(574fa66, BF16 unless marked FP32). R1 (pre-commit run --all-files) ran on the non-GPU Linux container sharing the same disk and clean git trees; R2–R4 and P1 ran on the GPU-attached container. Upstream changes from444638b..c338aaa(autopacked_prefill, fused Triton graph KV writes, and synchronousSpeculativeRunnerlayout refactoring) are covered under defaultpacked_prefill=None(effectiveTruein R3/P1), except where FP32 shared-memory limits required explicitpacked_prefill=False(disclosed in R4 below).444638b..740a097&c082bc9): The November 15 primary campaign (m7-gpu-validation-20261115-summary.zip, driver595.71.05) and November 7 independent-seed replication (m7-gpu-validation-20261107.zip, drivers595.71.05/580.76.05) are preserved below as full-campaign evidence prior to upstream autopacked_prefill.Test Result
Current rebased head (
c338aaa..c9dcdff, 2026-11-16)Functional, intersection regressions, and BF16 greedy equality
pre-commit run --all-files(Ruff lint, Ruff format, CPU regression tests)-m gpu --run-gpu)35 deselected,51.02 s)--deselect)97 deselected,105.24 s); includestest_cuda_graph.py::test_triton_graph_fuses_unmasked_key_and_value_writes[dtype0/1/2](3/3 PASSED)tests/test_speculative.py -m gpu -k "not flash_attn")100 deselected,26.11 s)c338aaavsc9dcdff, 10 GSM8K questions × 128 tokens)c=1andc=4; neutral-cwd probes verified effectivepacked_prefill=Trueon both treesThe 3 skips in the G5 Triton run come from single-GPU / missing optional dependency constraints (
1×two-GPU reservation,2×missingnixl), matching the prior campaign.Rebased FP32 5k distribution re-verification (
offset=2e6,nospec=3e6,STRESS=1)packed_prefill=False): Withc338aaa's defaultpacked_prefill=None,StageEngineautomatically enablespacked_prefill=Truefor CUDA + Triton regardless of dtype. OnOuro-1.4B(head_dim=128) in FP32,vllm_rlt/kernels/triton_attention.py:327(BLOCK_M=64, BLOCK_N=64, num_stages=2) requests114,816 Bshared memory, exceeding the RTX 5090101,376 Bhardware limit (triton.runtime.errors.OutOfResourcesinpaged_prefill_attention). Minimalsamples=1reproducers confirmed identical crashes on unmodified basec338aaaand candidatec9dcdff(diagnostics/fp32-nospec-s1-{base,cand}.log), showing this is an inherited upstream PR [Engine] Packed prefill resolves automatically, for Huginn too #149 FP32 sizing boundary rather than a rebase regression. To run thec338aaa..c9dcdffcomparison under the same unpacked prefill path used on 11/15, R4 was executed via an external driver settingExecutionConfig(packed_prefill=False)without modifying repository source or frozen scripts (DRIVER: packed_prefill=Falselogged for all 5 groups).c338aaa..c9dcdff, 5,000 seeds/group,α = 0.01,packed_prefill=False)t1controlnospec/ speculative 4-group)0.1976/0.6870(outside=0, all 5 groups PASS)asynct2goodness-of-fit0.2068(PASS;3,952 / 5,000,6strata,excluded=0)asyncvsnospect2two-sample homogeneity0.7473(PASS)t2aggregate GOFnospec/sync/async/sync_d1/async_d1)0.5971 / 0.5681 / 0.2068 / 0.8194 / 0.5069asyncvssynct2homogeneityasync_d1vsnospec0.3455/0.5097sync/async;sync_d1/async_d1)0.8053 / 0.8057;0.4969 / 0.5237(accepted_match=True,committed_match=True)R4-verdict.json)pass(verdict="pass")Rebased BF16 greedy non-regression (
P1, basec338aaavs candidatec9dcdff)Ouro-1.4B BF16, 64 tokens, ABBA (
base-1, cand-1, cand-2, base-2,n=4measured reps across 2 processes/side), defaultpacked_prefill=None(effectiveTrueverified on both trees):ctok/s(mean ± SD [min, max])tok/s(mean ± SD [min, max])tok/smsms132.783 ± 1.039 [31.415, 33.745]33.165 ± 0.938 [31.797, 33.809]+1.16%30.299 ± 0.970 [29.421, 31.592]29.956 ± 0.891 [29.355, 31.261]-1.13%3138.982 / 3138.982184/196 (93.9%)/184/196 (93.9%)no_regression4132.770 ± 1.232 [131.679, 134.180]129.021 ± 3.129 [125.431, 132.365]-2.82%29.882 ± 0.284 [29.563, 30.142]30.772 ± 0.779 [29.941, 31.668]+2.98%3547.710 / 3547.710736/784 (93.9%)/736/784 (93.9%)no_regressionBoth concurrencies satisfy the frozen F1 rule (
ranges_overlap=Trueon bothtok/sandTPOTatc=1andc=4; atc=4, ranges overlap over131.679–132.365 tok/sand29.941–30.142 mswithin run-to-run variance, and accepted drafts and peak allocated memory are identical).flash_attn_4/ non-SM120 GPUs, PD/NIXL/multi-GPU, and BF16 distribution checks.Code quality
Reviewed
444638b..740a097and direct callers against R01–R25 of.agents/skills/rlt-refactor/SKILL.md. No runtime refactoring issue was found in state ownership, device residency, result application, or asynchronous cleanup; one P3 failure-result docstring mismatch (AcceptResultinvllm_rlt/worker/speculative_sampling.py) was corrected and fixup-squashed into21a412d(740a097). Rebasing ontoc338aaaproducedc9dcdffwith an identicalgit range-diff(=on all 10 commits).Prior-base full validation (
444638b..740a097&c082bc9,git range-diffall=)Full 11/15 primary campaign and 11/07 independent replication (functional, #153 43-case integration, GSM8K smoke, 20k distribution, P1–P4, ops-only/full profiling, and c=1 timeline)
1. Functional,
#153integration, and base FA4 checks (444638b..740a097&c082bc9)-m "not gpu")740a097); 976 passed, 35 skipped (11/07env)444638b740a097production +5110f42tests)444638bvs candidate (10 questions × 128 tokens)c=1andc=4(both 11/15 and 11/07)The 61 CUDA cases cover two in-flight rounds, real CUDA Graph capture/replay, ragged and
K=0rounds, EOS/abort/ID reuse, numerical-error cleanup, and request-local RNG isolation. A layout regression verifies that when a non-argmax draft is accepted, the CPU committed sequence, next-round GPU input, and serial KV replay remain identical.2. GSM8K smoke and FP32 distribution (5k + 20k) (
444638b..740a097)GSM8K smoke: 10 fixed questions × 3 seeds (
{1, 2, 3}), 512-token limit,temperature=0.7, top_p=0.95. Both runs produced flexible-extract scores ofnospec22/30,sync24/30, andasync20/30 (strict-match12/30,13/30,12/30), with 1 length-truncated sample onasync(id=150, seed=2). Async vs sync had 3 wins / 7 losses / 20 ties (exact McNemarp = 0.344); async vs nospec had 4 / 6 / 20 (p = 0.754). No paired comparison reached significance; this small smoke only checks against large regressions.Distribution diagnostic (FP32, prior to auto
packed_prefill): One fixed prompt,temperature=1.0, top_k=20,K=3, draft depths1and2. Reference distributions use serial replay with the existingSampler.probabilities.nospecuses disjoint seeds. Atα = 0.01,t1is a prerequisite control and botht2primary criteria must pass:offset=2e6)offset=4e6, disjoint)offset=0)offset=0, includes 5k)t1controlnospec/ speculative 4-group)0.198/0.6870.125/0.3460.728/0.1800.313/0.0355t1out-of-support /chi2_excluded0/False0/False0/False0/Falseasynct2GOF0.2070.7520.5000.324asyncvsnospect2homogeneity0.7470.9800.3710.258t2aggregate GOFnospec/sync/async/sync_d1/async_d1)0.597 / 0.568 / 0.207 / 0.819 / 0.5070.398 / 0.391 / 0.752 / 0.841 / 0.1450.524 / 0.326 / 0.500 / 0.379 / 0.2740.420 / 0.497 / 0.324 / 0.216 / 0.903asyncvssynct2homogeneity0.3460.5890.3850.135async_d1vsnospect2homogeneity0.5100.3230.0340.878asynct2sample coverage (>= 200samples/stratum)6strata /3,952(79.0%)14strata /19,192(96.0%)6strata /3,905(78.1%)14strata /19,167(95.8%)t3aggregate GOFnospec/sync/async/sync_d1/async_d1)0.968 / 0.238 / 0.878 / 0.801 / 0.0760.625 / 0.797 / 0.391 / 0.279 / 0.458d2/d1)0.0851/0.35340.0856/0.35410.0863/0.35730.0856/0.3537sync/async;sync_d1/async_d1)0.805 / 0.806;0.497 / 0.5240.805 / 0.806;0.503 / 0.5190.809 / 0.808;0.502 / 0.5120.807 / 0.808;0.504 / 0.521t3strata: In the 11/15 20k run, all five group-levelt3aggregates pass (p = 0.076–0.968). The isolatedasync_d1t3stratum(t1, t2) = (26764, 13971)that hadp = 0.0068in the 11/07 20k run did not replicate in the 11/15 independent 20k run (p = 0.1734).chi2_excluded_samples=0counts only out-of-support exclusions; strata below 200 samples are reflected in the coverage row above.3. Performance, memory,
c=1timeline, andops-only/fullattribution (444638b..740a097)P1, base444638bvs candidate740a097, Ouro-1.4B BF16, 64 tokens, ABBAn=4measured runs across 2 processes/side): Committed throughput was25.63 ± 1.69 [24.16, 27.10] → 26.95 ± 0.25 [26.68, 27.18] tok/s(+5.14%) atc=1and100.91 ± 7.26 [94.54, 107.26] → 105.96 ± 0.86 [105.11, 106.82] tok/s(+5.01%) atc=4; TPOT was38.78 ± 2.56 [36.55, 41.00] → 36.76 ± 0.34 [36.45, 37.12] ms(-5.21%) atc=1and39.40 ± 2.85 [36.91, 41.88] → 37.38 ± 0.31 [37.07, 37.69] ms(-5.13%) atc=4. In the 5-stepops-onlywindow (c=1), base and candidate executed the exact same69,665kernels and35copies (69,700total GPU events,184.78 mskernel CUDA time /184.80 mstotal GPU activity time on both), with identical draft acceptance (184/196 = 93.9%inP1,46/49in the profiled request) and identical peak allocation (3,140.27 MiBatc=1,3,552.86 MiBatc=4).fullprofile ofrandom_async(c=1,188.87 mskernel CUDA time /189.11 mstotal GPU activity time),speculative_sampling.py:8 batched_probabilitiesaccounts for2.965 msself CUDA time (~1.6%); by phase,draftsampling activity takes2.007 ms(525GPU events,1.1%) andacceptactivity takes0.311 ms(220GPU events,0.2%). Peak allocated memory delta (random_asyncvsgreedy) is+9.74 MiBatc=1(3,150.00vs3,140.27 MiB;28,398vs28,388 MiBunder auto-KV) and+39.52 MiBatc=4(3,592.38vs3,552.86 MiB;28,219vs28,179 MiBunder auto-KV), within the existing36.3 / 145.0 MiBspeculative scratch reserve (11/07 transient allocation delta was11.9 / 48.2 MiB).flash_attn_4/ non-SM120 GPUs, PD/NIXL/multi-GPU, and BF16 distribution checks.Single-request (
c=1) greedy A/B timeline (base444638bvs candidate740a097, 64-token request, total trace span~3.93 s; zoom panel aligned atstep2, the first speculative decode round):4. Unprofiled BF16 measurements (
m7-gpu-validation-20261115/logs/G2-summary.md, 64 tokens,n=4measured reps across 2 processes/side)(Recorded as observations without speed claims across sampling modes; short-request random throughput reflects stochastic draft acceptance counts and round counts.)
ctok/s(mean ± SD[min, max])tok/s(mean ± SD[min, max])tok/smsms444638b) vs Cand greedy (740a097)125.63 ± 1.69 [24.16, 27.10]26.95 ± 0.25 [26.68, 27.18]+5.14%38.78 ± 2.5636.76 ± 0.343140.27 / 3140.27184/196 (93.9%)/184/196 (93.9%)444638b) vs Cand greedy (740a097)4100.91 ± 7.26 [94.54, 107.26]105.96 ± 0.86 [105.11, 106.82]+5.01%39.40 ± 2.8537.38 ± 0.313552.86 / 3552.86736/784 (93.9%)/736/784 (93.9%)asyncvs randomsync121.10 ± 0.44 [20.68, 21.48]23.89 ± 2.95 [21.05, 26.57]+13.25%47.21 ± 0.9842.00 ± 5.213150.00 / 3131.35168/240 (70.0%, 84 rds)/184/196 (93.9%, 68 rds)asyncvs randomsync478.38 ± 1.19 [76.70, 79.48]84.27 ± 7.17 [77.71, 90.54]+7.52%50.13 ± 0.7846.10 ± 3.943592.38 / 3529.33692/908 (76.2%, 316 rds)/720/832 (86.5%, 288 rds)asyncvsnospec(ordinary async)121.10 ± 0.44 [20.68, 21.48]16.82 ± 0.05 [16.77, 16.89]-20.29%47.21 ± 0.9859.47 ± 0.183150.00 / 3140.13168/240 (70.0%)/n/aasyncvsnospec(ordinary async)478.38 ± 1.19 [76.70, 79.48]64.55 ± 0.67 [63.57, 65.10]-17.64%50.13 ± 0.7861.96 ± 0.663592.38 / 3552.17692/908 (76.2%)/n/aasyncvs Candgreedy121.10 ± 0.44 [20.68, 21.48]27.03 ± 0.07 [26.96, 27.11]+28.14%47.21 ± 0.9836.65 ± 0.103150.00 / 3140.27168/240 (70.0%)/184/196 (93.9%)asyncvs Candgreedy478.38 ± 1.19 [76.70, 79.48]106.68 ± 0.12 [106.55, 106.83]+36.12%50.13 ± 0.7837.12 ± 0.043592.38 / 3552.86692/908 (76.2%)/736/784 (93.9%)5. Single-request (
c=1) full-request timeline summary (timeline/render.log, 64 tokens)ms)ms)444638b97.645,53518.0618.5%740a09795.32(-2.31)5,535(+0)18.03(-0.03)18.9%444638b3,830.58231,301614.4916.0%740a0973,835.79(+5.21)231,301(+0)614.49(+0.00)16.0%6. Scheduled 5-step (
steps 3..7)ops-onlyphase breakdown (logs/analysis-*-ops-only-c1.md)(Phase counts from
analyze_profile.pyinclude CUDA kernels plus device/pinned memcpy and memset events; forgreedy c=1, both base and candidate execute69,665kernels +35copies =69,700GPU events, with184.78 mskernel time /184.80 mstotal GPU activity.)c=1, 5 active steps)ms)draftEvents /msverifyEvents /msacceptEvents /mscollectEvents /msother(recurrent forward) Events /msgreedy / base(444638b)69,700(69,665+35)184.803(184.78kernel)0 / 0.0005,285 / 33.73875 / 0.09115 / 0.00764,325 / 150.967greedy / cand(740a097)69,700(69,665+35)184.801(184.78kernel)0 / 0.0005,285 / 33.78275 / 0.09115 / 0.00764,325 / 150.921random_async / cand(740a097)71,970(71,795+175)188.668525 / 1.997(1.1%)5,460 / 33.865(17.9%)220 / 0.310(0.2%)15 / 0.00665,750 / 152.490(80.8%)random_sync / cand(740a097)71,204(70,937+267)187.913826 / 3.099(1.6%)5,549 / 33.781(18.0%)66 / 0.166(0.1%)72 / 0.02464,691 / 150.843(80.3%)7. Scheduled 5-step (
steps 3..7)fullmode source-level stack attribution (logs/analysis-greedy-cand-full-c1.md,logs/analysis-random_async-cand-full-c1.md)vllm_rltLocation (file:line func)greedy / full / cand c=1Self CUDAms(Events)random_async / full / cand c=1Self CUDAms(Events)async_speculative.py:772 submit180.076(291,467)184.269(300,069)executor.py:165 forward179.373(288,607)179.318(288,609)async_speculative.py:449 _execute_recurrent_step177.351(289,972)177.274(289,974)ouro.py:283 recurrent_prepared177.006(288,362)176.930(288,364)ouro.py:210 forward176.200(285,612)176.123(285,614)stage_engine.py:740 _step_async_speculative143.966(233,209)147.292(240,091)async_speculative.py:598 _draft136.191(231,585)138.598(236,232)async_speculative.py:668 _verify43.786(59,152)44.839(60,217)normalization.py:20 forward44.105(121,257)44.176(121,244)rotary_embedding.py:14 apply_rotary_pos_emb16.447(43,202)16.488(43,203)kv_cache_manager.py:663 _write_prepared6.308(25,920)6.346(25,909)speculative_sampling.py:8 batched_probabilities0)2.965(3,810)async_speculative.py:588 _probabilities0)2.965(3,810)activation.py:20 silu_and_mul_reference2.657(4,800)2.656(4,800)Evidence
226 KiBbee7d5ee3b49b5211f6fe177e09040d7e857f5f1e05df17ef7547264684aff38c338aaa..c9dcdff): R1pre-commit, R2 G1/G5/sync-spec CUDA logs, R3 greedy A/B +packed_prefill=Trueprobes, R4 FP32 5k + SM120 smem diagnostic, P1 ABBA JSON/summary, and 50-entrySHA256SUMS21 MBc9cd4dd80bb755c6be844038388a8b1441117ff4c4fb87f13505dac932ae5d9c444638b..740a097), JSON/CSV, scripts,timeline-greedy-c1.svg, 13 profile checks and summaries, 6 stack files, 7 analyses, and 295-entrySHA256SUMS1.4 MBa88e24e4de5f388fdc45063b6fc86393169e8b166945ed93de8e84526c4734a4444638b..c082bc9), distribution/GSM8K/greedy JSONs, scripts, and 152-entrySHA256SUMS