Shypd.aiThe largest open index of AI toolsMergoo is an open-source library for merging multiple LLM experts, allowing for efficient training of the combined large language models. It supports various merging methods like Mixture-of-Experts and Mixture-of-Adapters.
Trending
Also listed in
About
Mergoo is an open-source Python library designed to simplify the process of merging multiple Large Language Model (LLM) experts and then efficiently training the resulting merged LLM. It enables users to integrate knowledge from different generic or domain-specific LLM experts, supporting methods such as Mixture-of-Experts (MoE) and Mixture-of-Adapters (MoA). The library offers flexible merging for each layer and supports popular base models like Llama (including LLaMa3), Mistral, Phi3, and BERT. It is compatible with various trainers including Hugging Face Trainer, SFTrainer, and PEFT, and can run on CPU, MPS, and GPU devices. Mergoo allows for training choices ranging from only the Router of MoE layers to fully fine-tuning the merged LLM.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
