Shypd.aiThe largest open index of AI toolsLLaMA-MoE is an open-source Mixture-of-Experts (MoE) model series built from LLaMA, designed for continual pre-training. It offers lightweight models with multiple expert construction methods and gating strategies for efficient deployment and research.
Trending
About
LLaMA-MoE is an open-source project that provides a series of Mixture-of-Expert (MoE) models based on LLaMA and SlimPajama, optimized for continual pre-training. It aims to offer smaller, more affordable MoE models suitable for a wide range of research and deployment scenarios. Key features include lightweight models with activated parameters between 3.0-3.5B, multiple expert construction methods (Neuron-Independent and Neuron-Sharing), and various MoE gating strategies like TopK Noisy Gate and Switch Gating. The project also emphasizes fast continual pre-training with FlashAttention-v2 integration and dynamic weight sampling, alongside abundant monitoring items for gate load, importance, and loss. It provides quick start guides, installation instructions, and examples for supervised fine-tuning (SFT).
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
