Repository navigation
feat(jetspec): add causal parallel Draft model and forward-KL loss #39
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -131,6 +131,51 @@ def dflash_mask_mod(b, h, q_idx, kv_idx): | |
| return dflash_mask_mod | ||
|
|
||
|
|
||
| def _create_dflash_dense_mask( | ||
| anchor_positions: torch.Tensor, | ||
| block_keep_mask: torch.Tensor, | ||
| ctx_len: int, | ||
| block_size: int, | ||
| is_causal: bool, | ||
| ) -> torch.Tensor: | ||
| """Materialize the block mask for backends without FlexAttention (CPU). | ||
| Returns a boolean ``[B, Q, KV]`` mask where True means visible. Invalid | ||
| blocks see only their own draft query row so softmax stays finite; their | ||
| objective weights are already zero. | ||
| """ | ||
| num_anchors = anchor_positions.shape[1] | ||
| draft_len = num_anchors * block_size | ||
| kv_len = ctx_len + draft_len | ||
| device = anchor_positions.device | ||
| q_index = torch.arange(draft_len, device=device) | ||
| kv_index = torch.arange(kv_len, device=device) | ||
| q_block = q_index // block_size | ||
| anchor = anchor_positions.index_select(1, q_block) | ||
|
|
||
| is_context = kv_index < ctx_len | ||
| mask_context = is_context.unsqueeze(0).unsqueeze(0) & ( | ||
| kv_index.unsqueeze(0).unsqueeze(0) < anchor.unsqueeze(-1) | ||
| ) | ||
| is_draft = kv_index >= ctx_len | ||
| kv_block = (kv_index - ctx_len) // block_size | ||
| mask_draft = is_draft.unsqueeze(0).unsqueeze(0) & ( | ||
| q_block.unsqueeze(0).unsqueeze(-1) == kv_block.unsqueeze(0).unsqueeze(0) | ||
| ) | ||
| if is_causal: | ||
| q_offset = q_index % block_size | ||
| kv_offset = (kv_index - ctx_len) % block_size | ||
| mask_draft = mask_draft & ( | ||
| kv_offset.unsqueeze(0).unsqueeze(0) <= q_offset.unsqueeze(0).unsqueeze(-1) | ||
| ) | ||
| valid = block_keep_mask.index_select(1, q_block) | ||
| mask = (mask_context | mask_draft) & valid.unsqueeze(-1) | ||
|
|
||
| diagonal = torch.zeros(draft_len, kv_len, dtype=torch.bool, device=device) | ||
| diagonal[torch.arange(draft_len, device=device), ctx_len + q_index] = True | ||
| return mask | (~valid.unsqueeze(-1) & diagonal.unsqueeze(0)) | ||
|
|
||
|
|
||
| class DFlashModel(nn.Module): | ||
| """DFlash training wrapper. | ||
|
|
@@ -820,6 +865,31 @@ def _draft_backbone( | |
| else block_masks | ||
| ) | ||
|
|
||
| dense_attention_masks = None | ||
| if block_mask is None and direct_cp_metadata is None and device.type != "cuda": | ||
| dense_cache = {} | ||
| dense_masks = [] | ||
| for layer_id in range(self.draft_model.num_layers): | ||
| options = self._block_mask_options_for_layer(layer_id) | ||
| causal = bool(options.get("is_causal", False)) | ||
| if causal and set(options) - {"is_causal"}: | ||
| raise ValueError( | ||
| "CPU causal Draft attention does not implement these mask options" | ||
| ) | ||
| if causal not in dense_cache: | ||
| dense_cache[causal] = ( | ||
| _create_dflash_dense_mask( | ||
| anchor_positions, block_keep_mask, seq_len, | ||
| self.block_size, is_causal=True, | ||
| ) | ||
|
Comment on lines
+881
to
+884
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [P2] Reject packed inputs here or preserve document isolation. Packed input can reach this CPU path with SDPA or FlexAttention configured, but _create_dflash_dense_mask receives no ctx_doc_ids and permits every context position before the anchor. For ctx_doc_ids=[0]*8+[1]*8, anchors [2, 9], and block_size=5, the second block can attend to positions 0-7 from the first document. The CUDA mask's document-equality check excludes those positions, so the CPU path silently computes a different objective. Since packed execution is explicitly outside this core's supported scope, rejecting packed metadata before building the dense causal mask is sufficient. Alternatively, pass the document IDs through and implement the same isolation rule. Add a regression that checks either explicit rejection or invariance to changes in the preceding document. |
||
| if causal else None | ||
| ) | ||
| dense_masks.append(dense_cache[causal]) | ||
| if True in dense_cache: | ||
| dense_attention_masks = ( | ||
| dense_masks[0] if len(dense_cache) == 1 else dense_masks | ||
| ) | ||
|
|
||
| if direct_cp_metadata is not None: | ||
| from dataclasses import replace | ||
|
|
||
|
|
@@ -946,6 +1016,7 @@ def _draft_backbone( | |
| draft_position_ids=draft_position_ids, | ||
| context_position_ids=context_position_ids, | ||
| block_mask=block_mask, | ||
| draft_attention_mask=dense_attention_masks, | ||
| noise_embedding=noise_embedding, | ||
| direct_cp_metadata=direct_cp_metadata, | ||
| csa_metadata=csa_metadata, | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -436,6 +436,7 @@ def forward( | |
| draft_position_ids: torch.Tensor, | ||
| context_position_ids: torch.Tensor, | ||
| block_mask=None, | ||
| draft_attention_mask=None, | ||
| direct_cp_metadata=None, | ||
| csa_metadata=None, | ||
| csa_prepared=None, | ||
|
|
@@ -453,6 +454,8 @@ def forward( | |
| draft_position_ids: [B, draft_len] — position IDs for draft tokens | ||
| context_position_ids: [B, ctx_len] — position IDs for context tokens | ||
| block_mask: FlexAttention BlockMask for block-causal attention | ||
| draft_attention_mask: Boolean [B, draft_len, ctx_len+draft_len] mask | ||
| for the non-FlexAttention fallback (for example CPU tests). | ||
| """ | ||
| bsz, draft_len, _ = draft_hidden.shape | ||
| ctx_len = context_hidden.shape[1] | ||
|
|
@@ -954,10 +957,18 @@ def observe_fixed_backward(gradient): | |
| enable_gqa=True, | ||
| ) | ||
| else: | ||
| # Fallback: bidirectional attention (no mask) — expand KV for SDPA | ||
| # Fallback: SDPA with an optional materialized block mask (CPU). | ||
| k = _repeat_kv(k, self.num_kv_groups) | ||
| v = _repeat_kv(v, self.num_kv_groups) | ||
| attn_output = F.scaled_dot_product_attention(q, k, v, is_causal=False, dropout_p=0.0) | ||
| if draft_attention_mask is None: | ||
| attn_output = F.scaled_dot_product_attention( | ||
| q, k, v, is_causal=False, dropout_p=0.0, | ||
| ) | ||
| else: | ||
| attn_output = F.scaled_dot_product_attention( | ||
| q, k, v, attn_mask=draft_attention_mask.unsqueeze(1), | ||
| is_causal=False, dropout_p=0.0, | ||
| ) | ||
|
|
||
| if not use_cudnn_flex: | ||
| attn_output = attn_output.transpose(1, 2).contiguous() | ||
|
|
@@ -1011,6 +1022,7 @@ def forward( | |
| draft_position_ids: torch.Tensor, | ||
| context_position_ids: torch.Tensor, | ||
| block_mask=None, | ||
| draft_attention_mask=None, | ||
| direct_cp_metadata=None, | ||
| csa_metadata=None, | ||
| csa_prepared=None, | ||
|
|
@@ -1037,6 +1049,8 @@ def forward( | |
| } | ||
| if hasattr(self.self_attn, "dsa_layer"): | ||
| qsa_kwargs.update(dsa_metadata=dsa_metadata, dsa_layout=dsa_layout) | ||
| if draft_attention_mask is not None: | ||
| qsa_kwargs["draft_attention_mask"] = draft_attention_mask | ||
| draft_hidden = self.self_attn( | ||
| draft_hidden=draft_hidden, | ||
| context_hidden=context_hidden, | ||
|
|
@@ -1300,6 +1314,7 @@ def forward( | |
| draft_position_ids: torch.Tensor, | ||
| context_position_ids: torch.Tensor, | ||
| block_mask=None, | ||
| draft_attention_mask=None, | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Keep the new mask argument after Inserting Suggested fix
|
||
| noise_embedding: Optional[torch.Tensor] = None, | ||
| direct_cp_metadata=None, | ||
| csa_metadata=None, | ||
|
|
@@ -1392,6 +1407,9 @@ def forward( | |
| layer_block_mask = block_mask | ||
| if isinstance(block_mask, (list, tuple)): | ||
| layer_block_mask = block_mask[layer_id] | ||
| layer_draft_mask = draft_attention_mask | ||
| if isinstance(draft_attention_mask, (list, tuple)): | ||
| layer_draft_mask = draft_attention_mask[layer_id] | ||
| layer_metadata = ( | ||
| direct_cp_metadata[layer_id] | ||
| if isinstance(direct_cp_metadata, (list, tuple)) else direct_cp_metadata | ||
|
|
@@ -1424,6 +1442,7 @@ def forward( | |
| draft_position_ids=draft_position_ids, | ||
| context_position_ids=context_position_ids, | ||
| block_mask=layer_block_mask, | ||
| draft_attention_mask=layer_draft_mask, | ||
| direct_cp_metadata=layer_metadata, | ||
| csa_metadata=csa_metadata, | ||
| csa_prepared=csa_prepared, | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| # Copyright 2026 Ant Group Co., Ltd. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| # The JetSpec method is described in "JetSpec: Breaking the Scaling Ceiling of | ||
| # Speculative Decoding with Parallel Tree Drafting" (Hu et al., 2026, | ||
| # arXiv:2606.18394). This is an independent ASpec implementation: the retained | ||
| # DFlash block head already provides the selected-Target-layer feature fusion and | ||
| # dual-source KV injection; JetSpec fixes causal intra-block attention. No | ||
| # JetSpec source file is copied. | ||
| """JetSpec causal-parallel Draft head over frozen Target features.""" | ||
|
|
||
| from aspec.models.draft.dflash import DFlashConfig, DFlashDraftModel | ||
|
|
||
|
|
||
| class JetSpecConfig(DFlashConfig): | ||
| """Causal-parallel Draft configuration with an explicit block contract.""" | ||
|
|
||
| model_type = "jetspec" | ||
|
|
||
| def __init__(self, *, is_causal=True, block_size=16, **kwargs): | ||
| if is_causal is not True: | ||
| raise ValueError("JetSpec requires causal intra-block attention") | ||
| if isinstance(block_size, bool) or not isinstance(block_size, int) or block_size < 2: | ||
| raise ValueError("JetSpec block_size must be an integer >= 2") | ||
| architectures = kwargs.pop("architectures", ["JetSpecDraftModel"]) | ||
| if architectures != ["JetSpecDraftModel"]: | ||
| raise ValueError("JetSpec uses exactly one JetSpecDraftModel architecture") | ||
| kwargs["is_causal"] = True | ||
| super().__init__(architectures=architectures, block_size=block_size, **kwargs) | ||
| if self.ffn_type != "dense": | ||
| raise ValueError("JetSpec currently supports dense Draft FFNs only") | ||
| layer_types = list(getattr(self, "layer_types", []) or []) | ||
| if any(kind != "full_attention" for kind in layer_types): | ||
| raise ValueError("JetSpec currently supports full intra-block attention only") | ||
| if getattr(self, "mtp_conditioning", None) is not None: | ||
| raise ValueError("JetSpec does not use the native-MTP conditioning contract") | ||
|
|
||
|
|
||
| class JetSpecDraftModel(DFlashDraftModel): | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Apply the JetSpec causal mask on non-CUDA devices. On any non-CUDA device, The CPU tests in Pass a boolean mask with the same visibility rules as the CUDA mask to SDPA on this branch. Add a CPU forward test that checks draft positions cannot see other blocks.
|
||
| """DFlash block head with the JetSpec causal configuration.""" | ||
|
|
||
| config_class = JetSpecConfig | ||
|
Comment on lines
+39
to
+42
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [P2] Enforce the causal contract when the Draft model is called directly. Changing config_class does not make the inherited attention causal: DFlashAttention does not consult config.is_causal, and a direct JetSpecDraftModel.forward call without block_mask, draft_attention_mask, or Direct CP metadata falls through to SDPA with is_causal=False. Later draft positions can therefore affect earlier outputs even though JetSpecConfig requires is_causal=True. The training wrapper supplies the correct mask; this finding concerns the separately exposed Draft model API. Please either construct the appropriate mask or fail explicitly when the required visibility metadata is absent. The official reference head applies its causal configuration in the attention path. Add a direct-Draft regression that perturbs a later noise embedding and checks earlier outputs, or checks that the unmasked call is rejected. |
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,98 @@ | ||
| # Copyright 2026 Ant Group Co., Ltd. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| # Objective reimplemented from the public JetSpec paper (Hu et al., 2026, | ||
| # arXiv:2606.18394, Section 3.3 and Appendix D): temperature-scaled forward KL | ||
| # between the frozen Target and the causal parallel Draft, with optional | ||
| # DFlash-style depth weighting and the T_KD^2 gradient scale. Configuration, | ||
| # anchor sampling, masks, reductions and recovery stay in the ASpec DFlash path. | ||
| """JetSpec causal-parallel Draft training objective.""" | ||
|
|
||
| import math | ||
|
|
||
| import torch | ||
| import torch.nn.functional as F | ||
|
|
||
| from aspec.models.dflash import DFlashModel | ||
|
|
||
|
|
||
| class JetSpecModel(DFlashModel): | ||
| """DFlash block head with causal intra-block attention and forward KL.""" | ||
|
|
||
| def __init__(self, *args, distillation_temperature=1.0, **kwargs): | ||
| # Main JetSpec results use uniform per-position weights (gamma = 0). | ||
| kwargs.setdefault("loss_decay_gamma", 0.0) | ||
| super().__init__(*args, **kwargs) | ||
| config = getattr(self.draft_model, "config", None) | ||
| if not bool(getattr(config, "is_causal", False)): | ||
| raise ValueError("JetSpec requires Draft config is_causal=true") | ||
| if self.mtp_conditioning is not None: | ||
| raise ValueError("JetSpec does not use the native-MTP conditioning contract") | ||
| if self.loss_objective != "decay": | ||
| raise ValueError("JetSpec supports the decay/uniform weighting objective only") | ||
| if self.ce_loss_alpha != 0 or self.l1_loss_alpha != 0: | ||
| raise ValueError("JetSpec replaces CE/L1 with forward KL; set both alphas to zero") | ||
| if self.prediction_start != 1: | ||
| raise ValueError("JetSpec supervises the suffix after the anchor position") | ||
| if (isinstance(distillation_temperature, bool) | ||
| or not isinstance(distillation_temperature, (int, float)) | ||
| or not math.isfinite(float(distillation_temperature)) | ||
| or float(distillation_temperature) <= 0): | ||
| raise ValueError("JetSpec distillation_temperature must be positive and finite") | ||
| self.distillation_temperature = float(distillation_temperature) | ||
| if config is not None and hasattr(config, "block_size"): | ||
| if int(config.block_size) != self.block_size: | ||
| raise ValueError("JetSpec training block_size must match its Draft config") | ||
|
|
||
| @property | ||
| def uses_target_distribution(self) -> bool: | ||
| return True | ||
|
|
||
| @property | ||
| def uses_target_hidden_states(self) -> bool: | ||
| return True | ||
|
|
||
| def _block_mask_options(self) -> dict: | ||
| return {"is_causal": True} | ||
|
|
||
| def _block_mask_options_for_layer(self, layer_id: int) -> dict: | ||
| return {"is_causal": True} | ||
|
|
||
| def _training_loss( | ||
| self, | ||
| *, | ||
| logits, | ||
| draft_hidden, | ||
| target_ids, | ||
| last_hidden_states, | ||
| lm_head_weight, | ||
| safe_label_indices, | ||
| packed_context_positions, | ||
| sequence_length, | ||
| input_ids, | ||
| anchor_positions, | ||
| ): | ||
| if last_hidden_states is None: | ||
| raise ValueError( | ||
| "JetSpec forward KL requires Target last_hidden_states; set " | ||
| "inference.store_last_hidden_states=true in the run config" | ||
| ) | ||
| target_positions = (safe_label_indices - 1).clamp(min=0) | ||
| aligned_hidden = self._align_target_hidden_states( | ||
| last_hidden_states, | ||
| target_positions, | ||
| global_sequence_length=sequence_length, | ||
| packed_context_positions=packed_context_positions, | ||
| ) | ||
| with torch.no_grad(): | ||
| target_logits = F.linear( | ||
| aligned_hidden.detach(), lm_head_weight.detach() | ||
| ).float().reshape(-1, logits.size(-1)) | ||
| draft_logits = logits.float().reshape(-1, logits.size(-1)) | ||
| temperature = self.distillation_temperature | ||
| log_q = F.log_softmax(draft_logits / temperature, dim=-1) | ||
| p = F.softmax(target_logits / temperature, dim=-1) | ||
| kl = F.kl_div(log_q, p, reduction="none").sum(-1) | ||
|
Comment on lines
+91
to
+95
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [P1] Bound the full-vocabulary projection and KL allocations. This eagerly materializes full FP32 teacher logits, draft logits, log_q, and p, followed by the vocabulary-sized unreduced KL output. With batch=2, the default 512 anchors and block_size=16, and a Qwen3-sized vocabulary of 151936, each [16384, 151936] FP32 tensor is about 9.27 GiB; these four live tensors alone occupy about 37.09 GiB, before temporary buffers, the original logits, model activations, gradients, and optimizer state. Anchor positions and zero-weight or invalid positions are also projected and distilled before the base wrapper masks their losses. This creates a substantial OOM risk at the default anchor budget that the small-vocabulary smoke does not cover. Please select the active prediction rows and bound both head projection and KL work with token chunks, using recomputation/checkpointing or a fused loss where necessary to avoid retaining every chunk's full-vocabulary intermediates for backward. Chunking only after the complete logits have been allocated does not address the whole peak. Add a realistic-vocabulary forward/backward memory check, and verify that the bounded path preserves the dense objective and gradients on a small reference case. |
||
| # The paper's Eq. 6 keeps the standard T^2 gradient scale. | ||
| kl = kl * (temperature * temperature) | ||
| return kl, kl, logits | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reject packed inputs on the CPU dense causal mask path.
On non-CUDA devices, the dense mask at lines 876-879 omits document IDs. Packed inputs still reach this branch when the attention backend is not
fa4orcudnn_flex_attention. In that caseuse_direct_cpis false, so no direct CP path replaces the mask. The dense helper then allows every context position before each anchor, and a later block can attend to an earlier document. Forctx_doc_idsof[0]*8+[1]*8, anchors[2, 9], andblock_size=5, the second block sees positions 0–7 of document 0. The FlexAttention path blocks this case with a document equality check, but the CPU path has no equivalent.Reject packed input before the dense mask is built. The unpacked path is unchanged.
Suggested fix
Committable suggestion
Source: Path instructions