Shypd.aiThe largest open index of AI toolsmakeMoE is an open-source Coding & Development tool that provides a from-scratch implementation of a sparse mixture of experts language model. It is inspired by Andrej Karpathy's makemore project and is useful for those studying or researching language models.
Trending
About
makeMoE offers a from-scratch implementation of a sparse mixture of experts (MoE) language model, drawing inspiration from Andrej Karpathy's 'makemore' project. This open-source tool is designed for developers and researchers interested in understanding and building MoE models. It features significant changes from the original makemore architecture, including sparse mixture of experts instead of a solitary feed-forward neural net, top-k gating and noisy top-k gating implementations, and Kaiming He initialization. The project also incorporates expert capacity for more efficient training. While it maintains the dataset, preprocessing, and language modeling task of makemore (generating Shakespeare-like text), makeMoE provides a hackable PyTorch implementation emphasizing readability over raw performance, making it an excellent resource for learning and experimentation.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
