Shypd.aiThe largest open index of AI toolsRetNet is an open-source Coding & Development tool that provides a PyTorch implementation of the Retentive Network. It serves as a successor to the Transformer architecture for large language models, prioritizing correctness and readability.
Trending
About
RetNet offers a minimal, pure PyTorch implementation of the Retentive Network, designed as a successor to the Transformer architecture for large language models. This repository focuses on aiding scientific and technological understanding and advancement, with code prioritizing correctness and readability over optimization. Key features include single-scale and multi-scale retention across parallel, recurrent, and chunkwise paradigms, as well as a multi-layer retentive network with FFN and LayerNorm. It also supports a causal language model built on top of the retentive network. The implementation utilizes Microsoft's xPos for positional encoding, with an alternative complex value encoding available for specific use cases, though it requires higher precision and memory.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
