Shypd.aiThe largest open index of AI toolsMini-SGLang is an AI Frameworks & Infra tool that provides a lightweight yet high-performance inference framework for Large Language Models. It offers advanced optimizations like Radix Cache and Chunked Prefill for efficient LLM serving.
Trending
Also listed in
About
Mini-SGLang is a compact and high-performance inference framework specifically designed for Large Language Models (LLMs). It serves as a lightweight implementation of SGLang, aiming to simplify the complexities of modern LLM serving systems. With a codebase of approximately 5,000 lines of Python, it functions as both a capable inference engine and a transparent reference for researchers and developers. Key features include advanced optimizations such as Radix Cache for KV cache reuse, Chunked Prefill to reduce peak memory usage, Overlap Scheduling to hide CPU overhead, Tensor Parallelism for multi-GPU scaling, and optimized kernels like FlashAttention and FlashInfer for maximum efficiency. It supports online serving with an OpenAI-compatible API and an interactive shell mode for direct model interaction.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
