Shypd.aiThe largest open index of AI toolsMarlin is an Open Source AI tool that provides an FP16xINT4 LLM inference kernel. It achieves near-ideal ~4x speedups for LLM inference up to medium batch sizes of 16-32 tokens, making it suitable for larger-scale serving.
Trending
About
Marlin is an extremely optimized FP16xINT4 matrix multiplication kernel specifically designed for Large Language Model (LLM) inference. It aims to deliver close to ideal (4x) speedups for batch sizes up to 16-32 tokens, significantly outperforming prior work that typically achieves comparable speedups only at 1-2 tokens. This makes Marlin particularly well-suited for larger-scale serving, speculative decoding, and advanced multi-inference schemes like CoT-Majority. The kernel employs numerous techniques and optimizations, including organizing computation for efficient L2 cache usage, asynchronous global weight loads, double buffering for shared memory loads, and careful ordering of dequantization and tensor core instructions. It also reshuffles quantized weights and group scales offline for ideal access patterns and uses a "striped" partitioning scheme for good SM utilization across various matrix shapes. Marlin requires CUDA >= 11.8, an NVIDIA GPU with compute capability >= 8.0 (Ampere or Ada), and torch>=2.0.0.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
