Shypd.aiThe largest open index of AI toolsLongLLaMA is a large language model capable of handling long contexts, based on OpenLLaMA and fine-tuned with the Focused Transformer (FoT) method. It offers various models, including code and instruction-tuned versions.
Trending
Also listed in
About
LongLLaMA is a large language model specifically designed to manage and process exceptionally long contexts, up to 256k tokens or more. Built upon the OpenLLaMA foundation and enhanced with the innovative Focused Transformer (FoT) method, it allows language models to handle extensive inputs while training on shorter sequences. The FoT method uses contrastive learning to enable attention layers to access a memory cache, significantly extending the effective context length. LongLLaMA is available in several variants, including a 3B base model under an Apache 2.0 license, and instruction-tuned versions like LongLLaMA-Instruct-3Bv1.1. A LongLLaMA Code 7B model, based on Code Llama, is also provided for code-related tasks. The project offers inference code, instruction tuning, and FoT continued pretraining code, making it a valuable resource for researchers and developers working with large language models and context scaling.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
