Shypd.aiThe largest open index of AI toolsExLlamaV3 is an optimized quantization and inference library for running LLMs locally on modern consumer GPUs. It features a new EXL3 quantization format based on QTIP and supports flexible tensor-parallel inference.
Trending
Also listed in
About
ExLlamaV3 is an inference library specifically designed for running Large Language Models (LLMs) locally on modern consumer-class GPUs. Its headline feature is the new EXL3 quantization format, which is based on QTIP from Cornell RelaxML, allowing for efficient model conversion in a single step. The library supports flexible tensor-parallel and expert-parallel inference setups, and provides an OpenAI-compatible server via TabbyAPI for local or remote inference. It also includes features like continuous, dynamic batching, HF Transformers plugin support, speculative decoding, and 2-8 bit cache quantization. ExLlamaV3 aims to make advanced quantization techniques more accessible and less resource-intensive, enabling users to run large models like Llama-3.1-70B with minimal VRAM.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
