Fastllm

Visit Tool

fastllm is a Coding & Development tool that provides a high-performance, dependency-free inference library for large language models. It supports tensor parallel inference for dense and MOE models on various GPUs.

Claim this tool

1View

At a glance

Pricing

Open Source

Free tier

Yes

API

Yes

Skill level

Technical

About

What is fastllm?

fastllm is a high-performance, dependency-free inference library implemented in C++ for large language models. It supports both tensor parallel inference for dense models and mixed-mode inference for MOE (Mixture of Experts) models, allowing for efficient deployment even on GPUs with limited memory (e.g., 10GB+ for full DeepSeek R1 671B). The library boasts excellent compatibility, supporting a wide range of NVIDIA, AMD, and domestic GPUs, including older cards like P100 and MI50. Key features include FP8 inference on any GPU, dynamic batching, streaming output, and multi-card tensor parallel inference. It also supports CPU + GPU hybrid inference for MOE models and offers easy installation via pip for Nvidia and AMD GPUs, with source compilation options for other platforms.

Best used for

Ideal for developers who need to deploy large language models efficiently, optimize inference performance on various GPUs, and utilize mixed-mode inference for MOE models. Especially valuable for achieving high throughput with models like DeepSeek on single GPUs or older hardware, and for integrating LLM capabilities into applications.

Common actions

deploy large models

optimize LLM inference

run models on GPU

quantize models

accelerate AI models

face swappinggithub copilot"AI Agents"open-sourceworkflowsautomated workflowdeepfakelow-code/no-codecollaboration

Capabilities

Key features

High-performance LLM inference
Tensor parallel inference
Mixed-mode MOE inference
FP8 inference support
Multi-GPU compatibility
CPU+GPU hybrid inference
Dynamic batching

Target Audience

developer

Integrations

Not yet documented

Pricing & Plans

Open Source

Free

FAQs

What types of GPUs does fastllm support for inference?

fastllm offers broad compatibility, supporting NVIDIA GPUs from M40 to 5090, AMD cards like MI50 and 7900, and various domestic cards including Tianshu, Muxi, and Suoyuan. It also supports FP8 inference on any GPU with more than 10GB of VRAM.

Can fastllm perform inference on MOE models with a single GPU?

Yes, fastllm supports CPU + GPU hybrid inference for MOE models, enabling single-card inference for large models like DeepSeek R1 671B even with limited GPU memory. This allows for efficient deployment without requiring multiple high-end GPUs.

How does fastllm handle model quantization and precision?

fastllm supports various precision levels including FP16, BF16, FP8, INT4, and INT8. It can perform online quantization during inference or export pre-quantized models for faster loading. Users can specify data types for both dense and MOE layers.

Does fastllm provide an API for integration?

Yes, fastllm offers an API server with an OpenAI-style interface, allowing developers to easily integrate its high-performance LLM inference capabilities into their applications. It supports features like dynamic batching and streaming output.

What are the performance benefits of using fastllm?

fastllm delivers high performance, with examples like DeepSeek R1 671B achieving 20tps (original FP8) or 30tps (INT4 quantized) on a single GPU with dual-socket servers, and multi-concurrency reaching over 60tps. This is due to its C++ implementation and optimized custom operators.

Trending

Subcategories trending in Coding & Development

Open Source & Models DevOps & Infrastructure No-Code / Low-Code Testing & QA Backend & APIs Prompt Engineering

Trending

Explore

Browse AI tools by category

Content & Design Productivity & Business Coding & Development AI Agents & Automation Research & Education Wellness & Lifestyle Career Development Marketing & Growth Data & Analytics Customer Support & CX Finance E-commerce