Shypd.aiThe largest open index of AI toolsMInference is an AI Frameworks & Infra tool that accelerates inference for long-context LLMs. It uses approximate and dynamic sparse attention to reduce latency by up to 10x while maintaining accuracy.
Trending
Also listed in
About
MInference is a powerful tool designed to significantly speed up the inference process for long-context Large Language Models (LLMs). By employing approximate and dynamic sparse attention calculations, MInference can reduce inference latency by up to 10x during the pre-filling stage on an A100 GPU, all while preserving model accuracy. It supports processing million-token prompts and has been integrated into various LLMs like Qwen2.5 and LLaMA-3.1. The framework also includes MMInference for multi-modality models and SCBench for evaluating long-context methods from a KV cache perspective, offering comprehensive solutions for optimizing LLM performance.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
