Shypd.aiThe largest open index of AI toolslong-context-attention is an open-source research tool that provides a unified sequence parallel approach for long context LLM model training and inference. It combines DeepSpeed-Ulysses-Attention and Ring-Attention for improved performance and versatility.
Trending
About
long-context-attention, also known as Unified Sequence Parallelism (USP) or Hybrid Sequence Parallelism, offers a novel approach to training and inference for long context Large Language Models (LLMs). This open-source project synergizes the strengths of DeepSpeed-Ulysses-Attention and Ring-Attention, addressing their individual limitations. Ulysses-Attention is sensitive to the number of attention heads and less suitable for GQA/MQA scenarios, while Ring-Attention can be less efficient in computation and communication. LongContextAttention provides a more general, versatile, and performant solution. It supports various FlashAttention versions (v2, v3) and can even run without FlashAttention for NPUs. The tool includes functionalities for setting process groups, extracting local tensors, and offers different ring implementation types like 'zigzag' and 'basic'. It has been verified in Megatron-LM and applied in several other projects, providing a robust solution for researchers and developers working with long context generative AI.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
Also listed in
