Llm-Awq

Visit Tool

llm-awq is an AI Frameworks & Infra tool that provides activation-aware weight quantization for LLM compression and acceleration. It supports efficient and accurate low-bit weight quantization (INT3/4) for various LLMs, including instruction-tuned and multi-modal models.

Claim this tool

1View

At a glance

Pricing

Open Source

Free tier

Yes

API

Skill level

Technical

About

What is llm-awq?

llm-awq is a powerful tool for Activation-aware Weight Quantization (AWQ) designed for Large Language Model (LLM) compression and acceleration. It offers efficient and accurate low-bit weight quantization (INT3/4) for a wide range of LLMs, including instruction-tuned models and multi-modal LMs. Key features include AWQ search for precise quantization, a pre-computed AWQ model zoo for popular LLMs like Llama-1/2/3, OPT, and CodeLlama, and memory-efficient 4-bit Linear in PyTorch. The tool also provides an efficient CUDA kernel implementation for fast inference, supporting both context and decoding stages. It includes examples for 4-bit inference with instruction-tuned models like Vicuna and multi-modal LMs such as VILA, and supports chunk prefilling for faster multi-round Q&A. llm-awq has received the MLSys 2024 Best Paper Award and is integrated into various platforms like Google Vertex AI, Amazon Sagemaker Containers, and Hugging Face transformers.

Best used for

Ideal for developers and data scientists who need to optimize the performance and memory footprint of Large Language Models. Especially valuable for deploying LLMs and VLMs on resource-constrained edge platforms, accelerating inference speed, and reducing GPU memory usage for efficient serving.

Common actions

quantize LLMs

accelerate LLM inference

compress LLM models

optimize LLM deployment

run LLMs on edge devices

collaborationopen-sourcelow-code/no-codedeepfakeautomated workflowgithub copilotface swapping"AI Agents"workflows

Capabilities

Key features

Activation-aware weight quantization
Low-bit weight quantization (INT3/4)
Pre-computed AWQ model zoo
Memory-efficient 4-bit Linear
Efficient CUDA kernel
Chunk prefilling
Supports multi-modal LMs

Target Audience

developerdata scientist

Integrations

google-vertex-aiamazon-sagemakerhugging-face-transformersnvidia-tensorrt-llmintel-neural-compressorfastchatvllmhuggingface-tgi+ 1 more

Pricing & Plans

Open Source

Free

FAQs

What types of LLMs does llm-awq support for quantization?

llm-awq supports a wide range of LLMs, including instruction-tuned models and multi-modal LMs. Its pre-computed model zoo covers popular architectures like Llama-1/2/3, OPT, CodeLlama, StarCoder, Vicuna, VILA, and LLaVA, enabling efficient low-bit weight quantization for diverse applications.

How does llm-awq improve LLM inference speed?

llm-awq improves inference speed through efficient low-bit weight quantization (INT3/4) and an optimized CUDA kernel implementation. This allows for faster processing during both context and decoding stages, significantly accelerating LLM and VLM responses, especially on edge devices like the Jetson Orin.

Can llm-awq be used for multi-modal models?

Yes, llm-awq seamlessly supports large multi-modal models (LMMs) such as VILA and LLaVA. It enables efficient 4-bit inference for vision language models, as demonstrated by its integration with TinyChat for applications like multi-image inputs and video reasoning on various hardware.

Trending

Subcategories trending in AI Agents & Automation

Chatbots & Conversational AI General-Purpose Agents Workflow Agents Personal Assistants RAG & Document AI Voice Agents

Trending

Explore

Browse AI tools by category

Content & Design Productivity & Business Coding & Development AI Agents & Automation Research & Education Wellness & Lifestyle Career Development Marketing & Growth Data & Analytics Customer Support & CX Finance E-commerce