Shypd.aiThe largest open index of AI toolsGaLore is a Coding & Development tool that provides memory-efficient LLM training through gradient low-rank projection. It allows full-parameter learning with greater memory efficiency than common low-rank adaptation methods.
Trending
Also listed in
About
GaLore is a memory-efficient low-rank training strategy designed for large language models (LLMs), enabling full-parameter learning while consuming less memory compared to traditional low-rank adaptation methods like LoRA. This tool utilizes gradient low-rank projection, making it independent of specific optimizers and easily pluggable into existing training pipelines with minimal code changes. GaLore offers various optimizers such as GaLoreAdamW and GaLoreAdamW8bit, supporting both standard and per-layer weight updates. It includes benchmark scripts for pre-training LLaMA models on datasets like C4 and fine-tuning RoBERTa on GLUE tasks, demonstrating its practical application and efficiency for developers working with LLMs.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
