Shypd.aiThe largest open index of AI toolsattention_with_linear_biases is an Open Source & Models tool that provides code for the ALiBi method in transformer language models. It enables input length extrapolation and is detailed in the ICLR 2022 paper 'Train Short, Test Long'.
Trending
About
attention_with_linear_biases is a GitHub repository offering the implementation of the Attention with Linear Biases (ALiBi) method for transformer language models. This method, presented in the ICLR 2022 paper 'Train Short, Test Long,' allows models to be trained on shorter input sequences (e.g., 1024 tokens) and then perform inference on significantly longer sequences (e.g., 2048 tokens or more) without requiring fine-tuning. The repository provides code and models for conducting experiments, specifically on the WikiText-103 dataset. ALiBi simplifies the positional encoding process by adding a linear bias to each attention score instead of using traditional position embeddings, which can improve performance even in non-extrapolating scenarios. The implementation details, including removing position embeddings and setting up the relative bias matrix, are clearly outlined.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
