Shypd.aiThe largest open index of AI toolsKVCache-Factory is an AI Frameworks & Infra tool that provides unified KV Cache compression methods for auto-regressive models. It supports multi-GPU inference with large language models like Llama-3-70B-Instruct.
Trending
About
KVCache-Factory is a unified framework designed for KV Cache compression methods specifically for auto-regressive models. It offers support for multi-GPU inference, making it suitable for large language models such as Llama-3-70B-Instruct. The framework integrates various compression techniques including PyramidKV, SnapKV, H2O, and StreamingLLM, and is compatible with Flash Attention v2 and Sdpa Attention. It provides tools for performance visualization and supports inference on benchmarks like LongBench and Needle in a Haystack. KVCache-Factory is an open-source project, making it accessible for developers and researchers working on optimizing LLM inference.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
