Repository navigation
Conversation
The engine serves one sequence at a time; docs/DSV41_BATCHING.md scopes continuous batching in four milestones. This is M1's first piece: the step scheduler as a header-only, GPU-free component (src/dsv41/ds4_sched.hpp) that owns the order and the token budget the engine will follow: * running sequences take the batch first, round robin: a sequence still reading its prompt gets its next chunk, a read one decodes one token; * waiting sequences fill the rest, a chat question jumping a queued 40K-token agent prompt (STRATA_ADMIT_SHORT_FIRST's rule, FIFO otherwise); * the per-step token budget caps prompt tokens, so prefill never starves decode; * cancel/retire, and max_new retirement. No behaviour change: nothing calls it yet, and the engine still runs its single-sequence path. M1 wires it into ds4_main.cpp (one kernel call per entry), M2/M3 replace those calls with varlen attention and a batched MoE. tests/dsv41/sched_test.cpp covers the admission order (short first and FIFO), chunking a 5000-token prompt, the budget across three prompts, round-robin decode, max_batch, cancel and retirement; ds4_sched_test is the CMake target.
M2 of docs/DSV41_BATCHING.md needs per-sequence KV: the single-sequence engine keeps one ring per layer, so a batch cannot address two sequences' tokens. This adds the index arithmetic as a GPU-free header: * KvSeq: a sequence's block table over an arena of block_tokens-sized blocks; slot(pos) for the varlen kernel, grow() that appends blocks from a free list, release() back to it; * KvBatch: the arena and its sequences in LRU order; touch() grows a sequence's table, evicting the oldest other sequence when the arena is full (the conversation slots' and expert cache's order), close()/open(). No behaviour change: nothing includes it yet; M1's wiring and M2's kernels come next. tests/dsv41/batch_kv_test.cpp covers slot arithmetic (offsets inside a block, block crossing), two sequences not sharing a slot, arena exhaustion with LRU eviction, a lone sequence never evicting itself, release/reopen and the LRU order; ds4_batch_kv_test is the CMake target.
The serve loop took the next stdin line and ran it to completion, so a queued chat question waited behind the agent prompt already in front of it. The line reader gains try_next()/push_front(), and before a GEN/GENI runs the loop looks at what is already waiting: a shorter prompt goes first (STRATA_DS4_ADMIT_SHORT_FIRST, default 2048 tokens, 0 turns it off) - the server rule, now at the engine where M4's queue needs it. One request at a time is unchanged, and nothing calls the batch scheduler yet; verified by building and --pack-info only, the behavioural test needs the model (a restart window).
The batch needs one thing between the scheduler and the kernels: for every entry of a step, the arena blocks that run's tokens live in and the offset inside the first one. This adds it as a GPU-free component: * KvBatch/KvSeq gain advance() and blocks(), so the caller can move a sequence on and read its table; * BatchPlanner::build() walks a StepPlan, grows each sequence's table through KvBatch::touch (evicting the oldest other sequence when the arena is full, reported to the caller), and returns SeqRun entries: position, token count, blocks, offset - what a varlen attention launch and a batched MoE take; * an entry for a sequence the arena does not hold is an error, not a crash. Nothing calls it yet; M2's kernels and the serve loop's queue are the callers. tests/dsv41/batch_plan_test.cpp covers a decode run beside a 6-token prefill (two blocks), advancing positions across a block boundary, the eviction report and the unopened-sequence error; ds4_batch_plan_test builds and passes in CMake.
admit_short_first() looked at the lines queued at that moment and swapped a single shorter one in front; it now collects everything waiting, orders requests shortest first (stable, non-requests after them) and takes the first - one pass, no starvation change beyond arrival order among equals. --admit-order reads GEN/GENI lines from stdin and prints the order the serve loop would take, without loading the model; the window test showed the reorder only fires for requests already queued when a step starts (the second request arrived 0.2 s after the first was taken, so the long prompt ran first).
The low-cost compromise in docs/DSV41_BATCHING.md lets two requests take turns on the one model: a burst of decoded tokens, then slot_save() into host memory, then the next request's slot_load(). This adds the memory side as a GPU-free component: * StateBank::put/get/drop over byte vectors, bounded by bytes and count; * a get() refreshes the entry, eviction takes the oldest first (the same order the KV block tables and the expert cache use); evictions and their bytes are counted; * a state larger than the whole budget is refused rather than silently parked. Nothing calls it yet: the serve loop's burst integration is the next step and needs a model run to verify. tests/dsv41/state_bank_test.cpp covers put/get/drop, replace, LRU order after a get, the byte budget and the refusal; ds4_state_bank_test builds and passes in CMake.
The serve loop still runs one request at a time; the flag is parsed and recorded so the split of the decode loop (tok/got/produced/counters parked in a GenState, slot_save/slot_load between turns) can follow without another CLI change. Nothing reads it yet: decoding is byte-for-byte what it was.
The engine runs one request at a time; a request can still be split into short generations whose continuation reuses the conversation (the engine skips the shared prefix), so another request takes the engine between two chunks. This adds the bookkeeping, with no engine and no HTTP in it: * split(max_new, burst) -> the chunk sizes; * BurstCursor: the growing id list for the continuation, tokens produced, the EOS cut inside a chunk and the stop when max_tokens is reached. serve/server.py does not call it yet; wiring it into the generation path and testing it with three clients is the next step, and it needs a server restart (about 90 s of model load). tests: serve/test_burst.py (4 tests).
Proves the bookkeeping of the chunked path without a model: three requests, one burst each per round, strict interleave; a short request leaves the rotation when it finishes; the continuation carries the prompt plus the tokens generated so far. serve/server.py still sends one generate() per request; wiring this into run() is the next step and needs a server restart to test.
A pass may now be capped at STRATA_BURST tokens; when it hits the cap and the model did not stop, the reply continues with prompt = ids + generated (the engine reuses the shared prefix), and between passes the FIFO is free for another request - the interleaving the batching design calls B2. Off unless STRATA_BURST > 0; the continuation is gated by STRATA_BURST_CONTINUE=1 so a capped pass without it still ends the reply (what the server did before). Validated against the mock engine (no GPU): the same 40-token request returns 8 tokens with STRATA_BURST=8 STRATA_BURST_CONTINUE=0 and the full 40 with STRATA_BURST_CONTINUE=1, the production port 8080 untouched (health 200).
serve/server.py turns its batched path on from one INFO line (self.batch = int(self.info.get("batch_slots"))), and it
drives the engine with BGEN/BGENI, BADM, BT, BDONE, BSTOP and BYIELD - not the "BGEN behaves like GEN" shape the handoff
sketched. The dsv41 engine speaks all of them now:
BGEN[I] <slot> <max_new> [keys] <ids> the prompt is read and its first token sampled as a GEN 1 (PP / T / DONE on the
control lines), then BADM <slot> <1: the slot decodes the rest | 0: done>
BT <slot> <id> ... BDONE <slot> <produced> <finish> <ms> the slot's tokens; <produced> counts the admission's token
BSTOP <slot> that slot ends at its next token (BDONE ... cancel)
BYIELD <slot> accepted and dropped: this engine reads a prompt whole
--batch N is what prints INFO batch_slots=N; without it the engine is exactly the one-at-a-time engine the server ran
before, so the production config is untouched. One slot at a time until per-sequence KV (M2): the model holds one
conversation, so a slot's answer runs to its end before the next request is read. The win is the protocol, not
throughput.
Tests: ds4_batch_proto_test (the line split and when an admission goes on; no GPU), and tools/dsv41/batch_probe.py
against a live engine on the box - BGEN's tokens equal GEN's for the same prompt (12/12), BADM 0 for max_new 1, BSTOP
ends a slot with BDONE ... cancel, a slot after a cancelled one still works, a bad slot gets ERR BGEN. Through the
server on port 8099 (--batch 1): one request alone answers exactly as production (0.66 s vs 0.65 s), three clients at
once all answer with the reference prompt's byte-identical answer through BGEN, a max_tokens 1 request completes, and a
client that leaves mid-stream (BSTOP) does not disturb the next request.
With --batch N > 1 the serve loop runs the admitted requests in turn: the model holds one conversation, so a slot that is not decoding is set aside in host memory (Model::slot_save) and loaded back for its next burst of --slot-burst tokens (16 by default); its tokens stream as BT <slot> <id> and BDONE ends it as before. A prompt read, an admission and a solo GEN all set the current slot aside first; a finished slot's conversation stays in the model and the next admission parks it into the conversation folder, so a later turn still reuses its prefix. What it buys is the latency half of docs/DSV41_BATCHING.md: three clients' answer tokens interleave instead of queueing (8 handovers over 47 answer tokens, against 2 with one slot) while the aggregate stays put (4.13 s with one slot, 4.23 s with two, 120/60/40 tokens) - the same tokens go through the same kernels, so M2/M3 are still what raises throughput. A request with pictures cannot be parked and runs to its end; a prompt read still blocks the other slots (BYIELD is dropped). tools/dsv41/batch_probe.py grew the two-slot check: two admissions sent at once, each slot's tokens equal its GEN reference, their BT lines interleave (0001110000111100001111), and each slot ends with its own BDONE - all green on the box. Over HTTP with --batch 2, three clients interleave, the reference answer admitted through BGEN behind them is byte-identical to production, and a client that leaves mid-stream (BSTOP) does not disturb the next request.
…1#656) The server reserves a slot and sends BYIELD <slot> when a much shorter prompt waits for the control lines; the engine ignored it, so a 4k-token agent prompt held the engine for its whole read. With --batch N the prompt is now fed in 1024-token slices, and between two of them a pending BYIELD ends the read: the part read stays in the model (its tokens = where it stopped), the engine answers YIELDED <slot> <tokens> + DONE (and BADM 0 for an admission), and the server sends the same prompt again later - the read goes on from there, never from 0. Without --batch the one-at-a-time engine is untouched (one feed, no slices). Measured on the box (4,031-token prompt, a short request 1.2 s behind it, --batch 2 --slot-burst 8): the short request's first token at 0.77 s instead of 5.47 s; the engine's own line "the prompt read gave way after 1024 of 4031 tokens". The HTTP check with the sliced read still gives the reference answer byte-identical to production, three clients interleaving (8 handovers), and a client that leaves mid-stream handled. Still one thing at a time: between its slices the read holds the model, so the short request's later tokens wait for the long read to finish. Interleaving the read slices with the slots' bursts (the scheduler of the design's section 3) is the next step.
A read gives way to a shorter request (BYIELD), but between its 1024-token slices it still held the model: the short request got its first token early and its answer waited for the whole read. Between two slices the read's state is now set aside (Model::slot_save), the active slots get bursts for about half the slice's time, and the state is loaded back - so a 4k-token prompt is read while the other slot's answer finishes. Measured (4,031-token prompt, a short request 1.2 s behind it, --batch 2 --slot-burst 8): the short request's first token 0.77 s and its whole answer 2.59 s, against 5.70 s with neither mechanism and 5.98 s with the yield alone. The HTTP check stays green: the reference answer byte-identical to production, three clients interleaving, a client leaving mid-stream handled. A read whose state is over 32k tokens is not interleaved (the copies per slice would cost more than they give), nor a read with pictures: both stay on the straight read.
STRATA_DS4_SLOT_TRACE=1 now prints per burst the tokens decoded, the burst's ms and the switch's ms (park + load), and per read slice its size in MB, its park and its load - the numbers the tuning needs. --read-slice N sets how much of a prompt one feed reads (default 1024; a BYIELD lands between two slices). Measured on the box (three clients, --batch 2, short conversations): a switch costs ~4.4-4.5 ms (the park and the load of the state, a few hundred tokens here), so a burst of 8 tokens (~120 ms of decode) pays ~3.7% and the default 16 ~1.9%; the wall time does not move with the burst (4.1-5.1 s across runs, the scheduling of three short clients is noisy), while the interleaving drops (8 handovers at burst 8, 2 at burst 32). The switch's cost follows the state's size, so the context - not the kernels - is what caps the parked-state approach at chat-sized conversations.
The two-slot check is now general: with --slots N the probe sends N admissions at once, each against its own prompt, compares every slot's tokens with the GEN the same prompt gives, checks each BADM <slot> 1 and each BDONE <slot> <n>, and that the BT lines take turns. --slots 3 on the box is all green (INFO batch_slots=3, three admissions, the order 012012..., three BDONEs). Through the server with --batch 3 the three-client check is green too: the reference answer byte-identical to production, the engine's trace shows slots 0, 1 and 2 decoding, mean switch 4.99 ms.
…admissions at once and the server's three-client check is green with --batch 3 (engine trace: slots 0, 1 and 2 decoding, mean switch 4.99 ms).
batch_soak.py drives the test server with rounds of concurrent streaming requests and checks that every one finishes with an answer and that the server stays healthy. On the box (--batch 2 --slot-burst 8): 8 rounds, 32 requests, 0 failed, 4.1-6.1 s a round, health 200 after, both slots decoding, mean switch 3.07 ms, no engine ERR line and no server traceback. serve.test_server over the same tree: 352 tests, OK (275 s).
…xt turn A slot that finished an answer sets its state aside in its own bytes (up to 64k tokens), and a later admission whose prompt shares at least 16 tokens with it loads it back before the prefix logic runs - so the server's next turn of the same chat stops at one of that conversation's checkpoints and reads only the new tokens. The match is made across every slot's kept conversation: a re-rendered turn drops the thinking the engine produced, so the kept state can be longer than the new prompt and the server's token-level slot_held matches nothing. Seen on the box (three-client check, --batch 2): the engine takes back slot 0's conversation with 101 of the prompt's 118 tokens shared, slot 1's with 28 of 43; the turn and HTTP checks stay green (the reference answers byte-identical to production). A single client still runs through the solo path (the server keeps it there while it is alone), where the engine continues its state in place as before - so a two-turn single-client test shows no slot reuse, by design.
… a canary a round) batch_fuzz.py sends rounds of 2-4 concurrent requests with random max_tokens (1..80) and, one in three, a client that goes away mid-stream, then a canary request: a slot nobody released, or a stream that never ends, shows up as a canary that stalls. On the box (--batch 2 --slot-burst 8): 12 rounds, every round answered, the canary came back in 0.29-0.46 s, both slots decoded, no engine ERR line and no server traceback.
A slot's turn switch copied its whole state to the host and back (slot_save/slot_load): ~4.5 ms on the box's short conversations. With STRATA_DS4_PARK_DEVICE=1 the state is parked in per-device arenas instead (Model::park_save / park_load: the same segments slot_save walks - every window ring, the compressors' partial groups and each source's compressed KV and index keys, per device and per tensor-parallel member - copied device to device, with the tokens and the checkpoints kept beside them). The host path stays the default and is untouched. On the box (--batch 2 --slot-burst 8): the two-slot probe passes every check with the device park, the three-client HTTP check is green (reference answer byte-identical to production), 0 host fallbacks and 0 load failures, and the engine's own trace goes from ~4.5 ms to 2.62 ms a switch. What is left in the 2.6 ms is the stream sync plus the checkpoints' host copies, which a move-based park would halve.
The design's open question was whether DSpark drafting should stay on when several sequences share the model ("likely
off"). The switch to test it is now there: STRATA_DS4_BATCH_NOSPEC=1 decodes a batch slot without the drafter (a solo
request keeps its drafts).
Measured on the box (--batch 2 --slot-burst 8, the three-client workload, two runs each): drafting on 4.27 s and 4.39 s,
drafting off 5.09 s and 5.31 s - 15-20% slower, with the same interleaving (8 handovers). So the answer is the opposite
of the guess: keep the drafter on in a batch at this size. The flag stays for the next measurement (more slots, longer
answers).
Measured on the box (batched engine, greedy, one request at a time): decode is 79 tok/s at a 216-token context, 90 at 3.5K, 90 at 10K and 77 at 20K - a few percent across a 100x context change; prompt reads settle at 585-590 tok/s from 3.5K on. The split-KV attention kernel's own header documents ~125 us a query, ~1% of a 12.5 ms step, so even at 20K the attention path is a few percent of the token. That re-orders the remaining work honestly: M2 (varlen attention) is what a *batched* step needs - each token's attention with its own KV and positions - but it is not the speedup; M3 (batched MoE) is where the 2-3x lives. The numbers and the engine log lines are in docs/DSV41_BATCHING.md's new section and the handoff.
…t 2-3x The engine already writes its routing (DS4_TRACE=file: a record a route request, u8 layer, u8 T, T x k_exp int16 expert ids). tools/dsv41/route_overlap.py parses it, groups the decode records into tokens by their layer cycle and compares the two slots' expert sets step by step. On the box (--batch 2 --slot-burst 1, two unrelated prompts): the two sequences share 25.5% of their six experts (1.53 of 6); consecutive tokens of one sequence share 22.1%. Batching two tokens therefore reads 10.5 expert rows instead of 12 (1.15x) and three read 14.9 instead of 18 (1.20x) - the design's 2-3x assumption does not hold on this workload, which re-bases M3: the throughput is in the reads, so it only pays when the overlap is much higher (closer prompts), and that is a scheduling question before it is a kernel one.
added 9 commits
October 27, 2026 19:18
…nish before the next pass stages The step drifted from the third token on at 4 slots and more, and only sometimes: the same 4-slot run with alternating prompts came out clean in 1 of 3, and a run with a per-layer sync (a debug check) came out clean while the unsynced ones drifted. Two buffers were reused before their readers were done with them, both inside a pass: * the window path's cross-group staging has no acknowledgements (the chunk path's x_ack is what the chunk path relies on), so a reader's copy into its table could still be in flight when the next pass's writer staged into the same slot: some rows then landed from another pass. run_dev now syncs a step's device at the end of its work - a pass a chunk, its copies done before the next pass stages - which is what the head device's sync already gave the logits. * the serial path (!wp.par, replicate) staged every row through the same pinned buffer (sr.h_ckv), so the next row's device-to-host copy could overtake the previous row's host-to-device copies; a row now stages in its own slice of it (h_ckv + t * ckv_rb(), h_ik + t * ik_rb()). After: the alternating-prompt probes are clean in 3 of 3 runs at 4 slots and 1 of 1 at 16, and 2 slots are exact against the bursts (96 tokens, 59.9 tok/s against 43.0). 16 slots with a different prompt a slot still drift (12 of 16 slots, from token 1-8 on) - that is a second, T-dependent bug, not this race; docs/DSV41_HANDOFF.md has the two 7/8-row boundaries it can be (the attention's staged kernel gives way to the packed one, and the routing mailbox's async limit) and the probe that separates them. Also: tools/dsv41/same_probe.py grew --alt (two prompts alternating, every even slot must answer slot 0 and every odd slot slot 1 - the probe that exposed the race) and --out; DS4_STEP_SERIAL=1 and DS4_STEPCHECK=1 are the debug switches the hunt used.
…ot a row region; the per-layer dump is the next probe
…a sync-route switch, and the comparison tool
…ug - layer 0..12 agree to 1e-7, layer 13 flips on a 1-ulp tie
…er the tokens nor the cost; exactness holds to 8 rows, rounding differs above
…K, the batch probe passed
…st and the grouped routing pays +58% at 8 slots
…d 8 slots 88.6 tok/s, both bit-identical to their burst references
suifei
changed the title
added 9 commits
October 27, 2026 21:51
… a slot) and M6 accepted on it
…prompts and their prefill blew the 300 s idle timeout
…ounts data events (an empty delta every 5 s), the live-mode read no longer stops the other slots past 32K, and the slot wait beats
…- the 300 s silence was in all three dialects, not just chat/completions
…oted into a slot keeps its conversation instead of re-reading the prompt (184 s -> 92 s on a 40K prompt)
… None put it back in the validation buffer) and a seat keeps its half-read state at a yield
…en: 572/634/1307 tok/s at read-slice 1K/8K/32K against 1649 for one stream
…alternating turn 35 s -> 1.9 s; a 233K cold read is 122 s at 1906 tok/s; the read-slice cost is the cards pipeline
added 2 commits
October 28, 2026 01:38
…ue too Problem: without --batch the single turn is strictly first come, first served, so a short request (a chat question, a health check, a tool's side request) waits behind every agent prompt that arrived before it - including the decode of a long answer - although STRATA_ADMIT_SHORT_FIRST already gives it priority in the batched queue. Result: with the same opt-in setting (unset or 0 keeps the old behaviour exactly), the requests waiting for the single turn are ordered the way StrataEngine._admit_next orders them: a prompt of at most STRATA_ADMIT_SHORT_FIRST tokens first, arrival order within the short and within the long group, and a longer one that has waited STRATA_ADMIT_MAX_WAIT_S counts as short, so a long request is never held back for good. The request that holds the turn is never interrupted - only the waiters' order changes - and a waiting request still sends its keep-alive every QUEUE_BEAT_S. Verification: * passed: serve.test_queue_keepalive, 6 tests (the 3 existing ones and 3 new: a late short request takes the turn first over three rounds, the age-out restores arrival order, and an ordered waiter still beats and drops out when its client leaves). The 3 new ones fail cleanly on main (no ordering). * failed, unrelated: the full serve suite has two failures - test_lifeline's watcher fails on unmodified code in this container, and test_responses.NoLeakedSampler fails 4 of 6 runs on unmodified main when it follows test_queue_keepalive (a leftover sampler thread). * not run: a real engine or a GPU; and the change with --batch, which uses its own ordering already. (cherry picked from commit d9569fd)
…ver.py sets SLOT_KEEP=1 for the engine); what changed is the 2-day expiry and the 0700 folder
suifei
marked this pull request as draft
October 27, 2026 18:11
added 3 commits
October 28, 2026 02:12
…validation on this box
… take-back claim of mine that was wrong
…k to one stream, 1M
|
Why was this closed? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
dsv41 batching: M1-M6, with the measurements the PR needs
Ready for review. M1, M2's foundations, M4, M5 and M6 are in, tested and measured; the single-launch decode step is
not, and section 5 says what that costs and what it still needs. The two test sections the owner asked for - concurrency
against the engine without batch slots, and how many concurrent requests the L20 holds at the production 1M context with
256K max_tokens - then the same two questions on Qwen3.8-Flash-Next (256K context, 128K max_tokens).
1. Concurrency against the engine without it (real 1M context)
Box: 4 x NVIDIA L20 (46 GB each), the IST-DAS 2.25 bpw pack, the production configuration (
--max-context 1048576,--experts 6,--tp 2, DSpark on). Every request asksmax_tokens: 262144(256K) and leaves after it has read itsshare; thinking is off, so the tokens counted are the answer's.
--batch): the server's FIFO--batch 3--batch)--batch 16The branch changes when a user sees their tokens: the stock engine runs the answers one after the other, so the last
client's first token waits for every answer before it (10.84 s at three clients, 25.93 s at sixteen), while the branch
admits them all at once (0.31-0.76 s at three, 4.15 s at worst at sixteen) and their tokens interleave. At three
clients the aggregate is identical (62.4 against 61.9 tok/s); at sixteen it is 12 % lower, because the slots' KV comes
out of the experts' VRAM. The aggregate itself needs the single-launch step (M2's remainder) and the grouped MoE (M3),
which the routing measurements bound at 1.15-1.28x on this box.
2. How many concurrent requests the L20 holds at 1M context (256K max_tokens)
Every run loaded the model at
--max-context 1048576and every client was answered.Sixteen concurrent requests at the 1M context load and are all answered (17 sequences: the sixteen slots and the
solo one), with 0.59 GB a GPU left. The price is expert residency: a sequence's compressed KV is ~0.54 GB a GPU
(~2.2 GB in all), taken from the experts - 4224 of 4224 experts stay in VRAM with one sequence, 3662 at 13 sequences and
3441 at 17 - and the aggregate drifts from ~62 to ~51 tok/s as the displaced experts are read from RAM. At a 32K
context the same sixteen slots keep every expert (4224 of 4224, the tables 0.30 GB against 9.14 at 1M) and score 57.2
tok/s, so about half of the drift is the displacement and half is what sixteen sequences' passes and working sets cost
by themselves. Capping what one slot may hold (the design's 32K a slot) is the lever that would raise both.
Why sixteen concurrent requests are 13.6 % slower in wall than the stock engine (31.08 against 27.35 s): the engine
says it at load - with one sequence "the experts VRAM leaves over (~0.0 GB) are read in place from the model's files",
with the sixteen slots' seventeen sequences "~13.8 GB". The 1M-context KV is 9.14 GB a GPU for them against 0.54 GB for
one, and the experts it displaces are read per token. The 32K control (all 4224 experts resident) takes 28.0 s for the
same sixteen clients - +2.4 % is the interleaving, ~+11 % the spill. Measured against the 31.08 s baseline:
--slot-burst 32STRATA_DS4_BATCH_NOSPEC=1--experts 5The three levers, in order: the single-launch step (amortise the weights - M2's remainder), a per-slot context cap (the
tables are what displace the experts), and
--experts 5where the last expert is not worth it. Warming helps only thefirst tokens (the pack's cold reads and the empty RAM store after a load), not the steady state.
3. Qwen3.8-Flash-Next: the same two questions (256K context, 128K max_tokens)
The Qwen path (
strata, ISTA-DASLab IQ2_XS, 69 GB) is untouched by this branch; it already decodes several requests inone window (
"parallel": N). Same box, same client,--max-context 262144and every request asksmax_tokens: 131072.It runs normally on this tree, and at 256K four requests are the sweet spot (all admitted at once, first tokens
within 1.87 s, 364 handovers) while one at a time keeps the highest aggregate (73.0 tok/s) at the price of the queue
(the third client waits 2.87 s) - and eight slots cost both (38.7 tok/s, 6.11 s) because the extra KV comes out of the
expert cache. The controls across contexts say the same thing from both sides: at 32K the same batching is a real
aggregate win (25.0 -> 45.6 tok/s, 1.8x, 377 handovers against 2) - what this branch's single-launch step (M2) and
grouped MoE (M3) would buy; at 1M the engine needs
--kv q4_0(with int8 the 1M-token KV leaves no VRAM for itsexpert tier and it refuses), where
parallel 1took 5.99 s for three clients (50.1 tok/s) andparallel 412.92 s forfour (31.0 tok/s, cache 68-78 %). The first case after each load pays the pack's cold reads, so its single-request rates
are the low ones.
4. What else the branch carries: the monitor's parallel view (M6)
The built-in API monitor (
/api-monitor) shows the batch as it runs, in the page's own vocabulary: a row of slot cellsthat light while a slot holds a request, an area chart of requests in flight over four minutes with the session's peak
marked, and four facts (in flight, slots busy, waiting, peak). It draws itself only where the server serves more than one
request at a time, so a single-request deployment looks exactly as it did.
The data is the engine's own -
/v1/status'sconcurrencygainedslots(a slot's busy flag),busyandwaiting(requests admitted but holding no slot). Measured with two slots and three clients at 8K:
in_flight=3 slots=BB busy=2 waiting=1, thenB.and..as they finish; the page carries the panel's seven elements and the three clients answeredin 5.16 s with 164 handovers.
5. Status: what is in this branch, and what is not
In, tested, and measured:
slots (
STRATA_DS4_BATCH_LIVE), and the scheduler that takes the model a slot a turn.AttnRowsin its explicit form, theper-row chunk path (
StepRowsthroughlayer_chunk), proved bit-identical to the contiguous path in process(the same chunk through both paths on the same state, 300 tokens in 38 chunks through every layer, no difference in the rows, the compressed KV or the index keys). Re-running that A/B on the final tree showed it had never reached the per-row branch:
--prefill-chunk 8is clamped to 256 at load andchunk_rowstakes the per-row path only at T <= 16, so both sides ran the same code. Driven at T = 16 the per-row path is ~270x slower than the contiguous one (a 128-token prompt: 857 ms against not finishing in 233 s); one cause is fixed here - the indexer's weight block was rebuilt once a row - and the rest is open. Two faults found on the way are fixed too: apos_rowsthat pointed at pageable host memory, and a per-row ring write that read the block after its own. Thedecode side's plumbing is in too:
Runcarries the descriptor,
forward_chunksbuilds one chunk of T rows from it, the Engram rows are read per row, andforward_rowsis the entry.--batch N, and the server's batched streaming path: three clients overHTTP, every one answered.
INFO batch_*(a sequence's tables, theroom left, the count it allows, the experts the cards hold, from the pack's geometry and
cudaMemGetInfo), the burstmeasured per turn from the switch cost, and a slot's context computed at load, written beside the pack and applied by
the next load (
STRATA_DS4_AUTO=1): at 17 sequences 1M a slot became 373K, the tables 964 -> 344 MB a sequence, theexperts 13190 -> 14809 of 15360 in VRAM.
Not in yet: the single-launch decode step itself - the serve loop calling
forward_rowsonce for all the active slotsand sampling each row (the pieces are in: the descriptor through the chunk path, the per-row Engram reads, the per-row
sampler). What it still needs is measured rather than assumed now: on this box every multi-device run is layer-split
(
--tp 2and--tp 4both collapse to one member a group), so a layer's compressed rows are written on another deviceand the per-row path refuses without a cross-group hand-off (the design note's item 2: one copy of the step's T latent
rows to the owning group, ~32 KB a layer a step); and the per-row chunk path is ~270x slower than the contiguous one at T = 16 (one cause fixed), so it needs a profile pass before the step can rest on it. Until the step lands the engine keeps one pass a sequence, so this branch buys waiting time and
fairness (first tokens within 4.15 s against 25.93 s at sixteen clients) rather than aggregate throughput. That step is also M3's
prerequisite: the grouped GEMM over the step's T routed rows is what would turn the 1.15-1.28x bound (
route_overlap)into the 2-3x the milestone promises.
dsv41: one pass for the whole batch
M2's batched step now runs on the decode path's kernels instead of the chunked prompt path: one pass over T rows, a
token of a sequence each, the same single launch a decode token takes. The chunk path paid ~200 ms of fixed cost a pass
(a device sync after the routing, the bounds over the bus, ~40 kernels a layer); the window path pays ~16 ms.
What changed
forward_rows->forward_step-> the window path'sforward_par/forward_serialandlayer(), with everyper-position piece taken from the row itself:
finish'spos_rowsrope, the row's own ring block in the windowbuffer, its own compressor group, its own rows of the compressed tables (one
finisha row, at the row's own positionand destination), and the indexer a row at a time - its own sequence's region of the index keys, and its own picks
made absolute in that region (
rows_add_iwithkv_base(sr, seq)).readers place them in their own copies, and every device's work is finished before the next pass stages.
finish'spos_rowsindexes the query, not the row (pos_rows[v / vpp]). A step'sqcarriesvpp = NHLrowsa query, so the old indexing read another row's position - and the chunk path's step wrote its compressed rows
unroped (
rd = 0), which is the drift it showed from the third token on.--slot-context N/STRATA_DS4_SLOT_CONTEXT: the tokens a conversation may hold. The pack's compressed tables aresized a sequence by it - 965 MB a sequence at 1M, 30 MB at 32K - which is what keeps the experts in VRAM: at 16
slots and 1M they cost 15.4 GB a GPU and 2170 of 4224 experts go to the host lane (a pass then reads ~27 GB); at 32K,
none do.
copy could still be in flight when the next pass staged), and the serial path staged every row through one pinned
buffer (
replicate); a pass's device is now synced at the end of its work, and a row stages in its own slice.Numbers
Four L20s, 1M max-context, greedy, DSpark off, 64-token prompts, 96 tokens a slot,
--slot-context 32768, each step runagainst the bursts of its own slot count (
batch_step_bench.py --ref):The step's first tokens all arrive at one instant (0.88 s at 2 slots, 3.40 s at 8, 6.75 s at 16 - the last prompt's
read) against the bursts' 0.41 s spread, and at 16 slots nothing goes to the host expert lane.
At 1M with no slot cap (a shorter run, 32 tokens a slot): 2 slots 31.3 -> 39.4 (+26 %, 125 host experts) and 8 slots
31.0 -> 49.1 (+58 %, 984 host experts, 2.6x a slot). The flow streams a step's experts on this path at no cost
(39.4 with it on, 39.5 with
DS4_FLOW=0, the same tokens), which the chunk path's flow could never do - it deadlockedon a step's rows.
Exactness
Bit-identical to the burst path at 2, 4 and 8 slots. Above 8 rows the pass is deterministic and self-consistent but
rounds differently from a one-token decode: in the first pass, layers 0-12 agree to 1e-7 and layer 13 jumps to 1e-2 (an
indexer top-k tie flipping on a 1-ulp difference), which greedy decode then amplifies. Not a race, not a per-row
defect - the harness is
tools/dsv41/same_probe.py(one prompt for every slot, or two alternating) pluscmp_hash.py/cmp_num.pyover the per-layer hashes of row 0 (DS4_STEPCHECK=2).Tests
serve.test_server: Ran 348 tests, OKtools/dsv41/batch_probe.py(two slots): all checks passedtools/dsv41/batch_step_bench.py,same_probe.py,steprows_check.pydocs/DSV41_HANDOFF.md: the whole path - measurements, the dead ends, and the two remaining questions.Debug switches the work left behind, all off by default:
DS4_STEP_CHUNK=1(the chunk path's step, for an A/B),DS4_STEP_SERIAL=1(a pass on one thread,replicate, no staging),DS4_STEP_SYNCROUTE=1(the route on the device,no mailbox),
DS4_STEPCHECK=N(pass shape, per-layer hashes, row-0-vs-row-2 check).