Shypd.aiThe largest open index of AI toolsZhiLight is an AI Frameworks & Infra tool that accelerates LLM inference. It is a highly optimized engine for Llama and its variants, offering significant performance advantages over mainstream open-source engines.
Trending
About
ZhiLight is a highly optimized open-source LLM inference acceleration engine developed by Zhihu and ModelBest Inc. It is specifically designed to accelerate the inference of models like Llama and its variants, particularly on PCIe-based GPUs. The engine boasts significant performance advantages compared to other mainstream open-source inference engines such as vLLM and SGLang. Key features include an asynchronous OpenAI compatible interface, custom tensor and global memory management, encode and all-reduce overlap using "dual streams," and support for various quantization methods like Int8, SmoothQuant, FP8, AWQ, and GPTQ. ZhiLight also supports dynamic batching, flash attention prefill, prefix caching, and a wide range of models including Llama, Mixtral, and Qwen2 series.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
