Shypd.aiThe largest open index of AI toolsmesh-transformer-jax is an open-source coding & development tool that provides model parallel transformers in JAX and Haiku. It is designed for scalability up to approximately 40B parameters on TPUv3s.
Trending
About
mesh-transformer-jax is a Haiku library that leverages JAX's xmap/pjit operators for model parallelism of transformers, offering a parallelism scheme similar to the original Megatron-LM. This design is optimized for efficiency on TPUs due to their high-speed 2D mesh network. The library also includes an experimental model version with ZeRo style sharding. It is built for scalability, supporting models up to approximately 40 billion parameters on TPUv3s. The project provides pre-trained models like GPT-J-6B, a 6 billion parameter autoregressive text generation model, and includes guides for fine-tuning. It is particularly useful for AI/ML researchers and developers working with large-scale transformer models.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
