Shypd.aiThe largest open index of AI toolsLongLoRA is an Open Source & Models tool that provides code and documents for efficient fine-tuning of long-context Large Language Models (LLMs). It includes models from 7B to 70B with context lengths up to 100k.
Trending
About
LongLoRA offers a comprehensive solution for researchers and developers working with long-context Large Language Models. The tool provides code and documentation for both LongLoRA and LongAlpaca, an instruction-following dataset. Key features include an efficient shifted short attention mechanism that is compatible with Flash-Attention and not required during inference. It supports various model sizes, from 7B to 70B, and context lengths up to 100k. LongLoRA also facilitates supervised fine-tuning, including support for QLoRA to reduce GPU memory costs, and offers pre-trained weights for LLaMA2 and GPTNeoX models. The project includes evaluation scripts for perplexity validation and tools for merging LoRA weights.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
