Shypd.aiThe largest open index of AI toolsGPTQ is an open-source code implementation for accurate post-training quantization of generative pretrained transformers. It allows compressing models from the OPT and BLOOM families to 2/3/4 bits, including an efficient implementation of the GPTQ algorithm.
Trending
About
GPTQ provides an efficient, open-source implementation of the GPTQ algorithm for accurate post-training quantization of generative pretrained transformers. This tool enables developers to compress large language models from the OPT and BLOOM families down to 2, 3, or 4 bits, significantly reducing their memory footprint and computational requirements while maintaining accuracy. Key features include support for weight grouping, evaluation of perplexity on various language generation tasks, and performance evaluation on ZeroShot tasks. The repository also offers a 3-bit quantized matrix full-precision vector product CUDA kernel and benchmarking code for individual matrix-vector products and language generation with quantized models. Recent updates include static groups options, adjusted preprocessing for C4 and PTB, optimized 3-bit kernels for faster generation, and a minimal LLaMa integration with new tricks like `--act-order` and `--true-sequential` for improved accuracy.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
