Fastformers

Visit Tool

FastFormers is an Open Source & Models tool that provides highly efficient transformer models for Natural Language Understanding (NLU). It offers methods and recipes to achieve significant speed-ups for inference on CPU and GPU.

Claim this tool

No Views Yet

At a glance

Pricing

Open Source

Free tier

Yes

API

Skill level

Technical

About

What is fastformers?

FastFormers is an open-source project from Microsoft that provides a collection of methods and recipes for achieving highly efficient inference with Transformer models, specifically for Natural Language Understanding (NLU) tasks. The tool demonstrates impressive speed-ups, including a 233x acceleration on CPU with multi-head self-attentive Transformer architecture. It allows users to replicate results presented in the FastFormers paper and supports various optimization techniques such as model training, distillation, pruning, 8-bit integer quantization for CPU with ONNX Runtime, and 16-bit floating point conversion for GPU. The repository is built on top of several open-source projects including Hugging Face's transformers and ONNX Runtime.

Best used for

Ideal for developers and data scientists who need to significantly speed up the inference of Transformer models for Natural Language Understanding, reduce computational resource requirements, and replicate research findings on model efficiency. Especially valuable for deploying NLU models in resource-constrained environments or for large-scale applications.

Common actions

optimize transformer models

accelerate NLU inference

quantize deep learning models

prune neural networks

"AI Agents"github copilotface swappingautomated workflowworkflowsopen-sourcedeepfakecollaborationlow-code/no-code

Capabilities

Key features

Efficient Transformer inference
Model distillation
Model pruning
8-bit integer quantization
16-bit floating point conversion
SuperGLUE benchmark support

Target Audience

developerdata scientist

Integrations

onnxruntimehugging-face-transformers

Pricing & Plans

Open Source

Free

FAQs

What are the system requirements for running FastFormers?

FastFormers currently supports Linux operating systems. For CPU, AVX2 and AVX512 instruction sets are required, with AVX512 recommended for full speed. For GPU, Volta or later architectures are needed to utilize 16-bit floating point speed-up. ONNX Runtime v1.8.0+ is also a requirement.

Can I use pre-trained Hugging Face models with FastFormers?

Yes, FastFormers is built on top of Hugging Face's transformers, allowing you to use their pre-trained models. You can specify models like 'distilroberta-base' for training and distillation. You can also modify config files to change activation functions for better inference speed.

How does FastFormers achieve such significant speed-ups?

FastFormers employs several optimization techniques including model distillation (task-specific), structured pruning (reducing heads and FFN dimensions), 8-bit integer quantization for CPU inference with ONNX Runtime, and 16-bit floating point conversion for GPU inference. These methods collectively contribute to highly efficient transformer model inference.

Trending

Subcategories trending in Coding & Development

Code Assistants DevOps & Infrastructure No-Code / Low-Code Testing & QA Backend & APIs Prompt Engineering

Trending

Explore

Browse AI tools by category

Content & Design Productivity & Business Coding & Development AI Agents & Automation Research & Education Wellness & Lifestyle Career Development Marketing & Growth Data & Analytics Customer Support & CX Finance E-commerce