Shypd.aiThe largest open index of AI toolsMuon is an open-source optimizer for hidden layers in neural networks, designed to improve training speed and efficiency. It significantly reduces the time and cost required to train large language models and other neural networks.
Trending
About
Muon is an open-source optimizer specifically engineered for the hidden layers of neural networks. It aims to enhance the training process by optimizing these hidden weights, while other parameters like embeddings and classifier heads are handled by standard optimizers such as AdamW. The tool is based on research described in a dedicated thread and writeup, offering a specialized approach to neural network optimization. Muon has demonstrated notable accomplishments, including lowering the record for CIFAR-10 training, achieving GPT-2 (XL) performance with reduced compute costs, and improving training speed for GPT-2 (small) performance by a factor of 1.35x. It has been utilized by frontier labs like Kimi.ai for scaled LLM training and is particularly effective for training with large batch sizes.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
