Shypd.aiThe largest open index of AI toolsMedusa is an open-source framework for accelerating LLM generation using multiple decoding heads. It significantly speeds up LLM inference, offering up to 3.6x faster generation with parameter-efficient training.
Trending
About
Medusa is an open-source framework designed to accelerate Large Language Model (LLM) generation by employing multiple decoding heads. It addresses common pain points of other acceleration techniques, such as the need for a good draft model, system complexity, and inefficiency with sampling-based generation. Medusa adds extra 'heads' to LLMs to predict multiple future tokens simultaneously, combining these options with a tree-based attention mechanism for faster decoding. The framework supports both Medusa-1 (training only new heads) and Medusa-2 (full-model training with a special recipe), and includes self-distillation for fine-tuned LLMs without original training data. It offers significant speedups, particularly for single-GPU inference with a batch size of 1.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
