Skip to content

Add Medusa multi-head draft training #17

Open

Description

@zhouaihui

Train multiple future-token prediction heads from Target features and export them for Medusa
proposal generation. This extends ASpec to a lightweight head-based Draft design.

Suggested scope

A heads-only Medusa-1 recipe with a frozen, already supported Target is a useful first scope.
Select the head design and training objective during implementation, following the chosen public
reference. Keep the model's prediction distances and document boundaries consistent with its
supervision.

Getting started

A first PR may implement the heads and loss, or a focused feature-extraction prerequisite.
Include checks using upstream code or an independent numerical reference. Recipe and asset
integration can follow in separate PRs.

Reuse an existing Medusa evaluator for candidate-tree acceptance where possible. A missing asset
adapter can be tracked separately from the initial training contribution.

Expected outcome

  • The selected model, supervision and frozen Target bindings are documented and checked.

  • A runnable recipe supports training, exact resume and export of the trained heads.

  • Exported heads load strictly and reproduce checkpoint predictions.

  • Per-head held-out observations and supported scope are recorded; any decode-equality claims
    are verified under the evaluator's acceptance policy.

  • Run a small training comparison with the chosen reference implementation, using the same
    starting weights, data, training budget and settings. Compare loss curves and acceptance lengths
    with the same evaluation setup, and explain any noticeable differences.

This comparison can come in a follow-up once training and evaluation are ready; it does not need
to be part of the first model/loss PR.

References

Activity

  1. sysuTobo commented on Nov 13, 2026

    @sysuTobo
    Contributor

    I'd like to work on this issue, starting with the heads-only Medusa-1 scope suggested above.

    Reference: FasterDecoding/Medusa @ e2a5d20c048a9b0a4092e6933c34313687422518 (medusa/model/medusa_model.py + medusa/train/train_legacy.py). I plan to preserve its exact semantics: K parallel heads, each medusa_num_layers zero-initialized ResBlocks (Linear+SiLU+residual) followed by a bias-free vocab projection; CE-only objective summed across heads; assistant-only supervision. Note the legacy trainer supervises head i at position t with the token at t+2+i — I will keep this offset consistent with the reference and document it explicitly, with numerical checks against the reference equations.

    Target: Qwen3-8B (already supported), frozen; features via the existing post-norm final-hidden-state path (the native-MTP single-slot convention), so no aux-layer capture is needed.

    First PR: the head model + loss as a standalone module with offline CPU numerical tests (forward/gradient parity against an independent reference, mask/document isolation, future-token leakage, zero-init identity), following the tests/cpu/test_dmtp.py pattern. Follow-up PRs would cover Plan/trainer integration, exact resume, native export with strict reloading, a recipe, and per-head held-out observations. The initial training scope would be CP1/non-packing. Medusa candidate-tree acceptance needs a consumer adapter that SGLang does not currently provide; per the issue guidance I would track that separately rather than expand this into a serving project.

    Happy to adjust scope or reference choice based on maintainer feedback.

  2. sysuTobo commented on Nov 17, 2026

    @sysuTobo
    Contributor

    Progress update: the heads-only model and objective core landed in #32 (FasterDecoding/Medusa @ e2a5d20, CE-only, head i supervised by token t+2+i, assistant-only, full vocabulary).

    Remaining expected outcomes are planned as two increments:

    1. Integration PR: algorithm registration, Plan/trainer wiring (single-slot post-norm feature contract, CP1/non-packing initial scope), a docs/user-guide/medusa.md method note following the P-EAGLE precedent, and the shipped draft config.
    2. Recipe/asset PR: runnable recipe, heads-only native export with strict reloading, per-head held-out observations, and the documented supported scope. Medusa candidate-tree acceptance still needs a consumer adapter; per the issue guidance this stays tracked separately.
  3. sysuTobo commented on Nov 17, 2026

    @sysuTobo
    Contributor

    Second increment is up as a draft: #41 registers the heads-only Medusa family and wires Plan, the single-slot post-norm feature contract, the producer hook, the trainer and the checkpoint topology. It also ships the Qwen3-8B K=5 draft config, a Store/TCP 4K example and docs/user-guide/medusa.md.

    Local CPU verification is complete (1047 passed incl. 63 Medusa tests; lint/docs/compileall pass). GPU validation is still pending: 2x L20 could not be scheduled on the shared cluster (multi-GPU L20 pressure) and the H200 pool is unavailable to this team, so the PR claims no GPU result yet. The remaining expected outcomes (export, a maintained recipe, held-out observations, acceptance) stay in the follow-up increment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions