Shypd.aiThe largest open index of AI toolsText Generation Inference (TGI) is an open-source AI framework that provides a high-performance server for deploying Large Language Models (LLMs). It enables optimized text generation with features like tensor parallelism and continuous batching.
Trending
About
Text Generation Inference (TGI) is an open-source toolkit designed for deploying and serving Large Language Models (LLMs) with high performance. Developed by Hugging Face, it's used in production for services like Hugging Chat and the Inference API. TGI supports popular open-source LLMs including Llama, Falcon, and BLOOM, offering features such as tensor parallelism for faster inference on multiple GPUs, token streaming, and continuous batching for increased throughput. It also includes optimized transformers code with Flash Attention and Paged Attention, various quantization methods (bitsandbytes, GPT-Q, AWQ, Marlin, fp8), and hardware support for Nvidia, AMD, Inferentia, Intel GPU, Gaudi, and Google TPU. While TGI is now in maintenance mode, it has influenced the development of other optimized inference engines like vLLM and SGLang, which Hugging Face now recommends.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
Also listed in
