FastV

Visit Tool

FastV is an Open Source & Models tool that provides plug-and-play inference acceleration for large vision-language models. It achieves significant FLOPs reduction by pruning redundant visual tokens in deep layers without harming performance.

Claim this tool

1View

At a glance

Pricing

Open Source

Free tier

Yes

API

Skill level

Technical

About

What is FastV?

FastV is an open-source inference acceleration method specifically designed for large vision-language models (LVLMs). It operates as a plug-and-play solution, significantly reducing computational costs by pruning redundant visual tokens in the deeper layers of these models. This approach allows for a theoretical FLOPs reduction of up to 45% without compromising performance. FastV has been accepted to ECCV 2024 as an Oral Presentation, highlighting its innovative contribution to the field. The project provides code for setup, visualization of inefficient attention over visual tokens, and comprehensive evaluation scripts for latency and performance reproduction. It supports HuggingFace LLaVA models and is compatible with KV Cache for improved efficiency, particularly in video understanding tasks.

Best used for

Ideal for developers and data scientists who need to accelerate the inference of large vision-language models, reduce computational overhead, and visualize attention mechanisms. Especially valuable for researchers and practitioners working with LVLMs and seeking efficient deployment solutions.

Common actions

accelerate AI models

optimize vision models

reduce computational cost

evaluate model performance

workflowsopen-sourcecollaborationautomated workflowlow-code/no-codedeepfake"AI Agents"github copilotface swapping

Capabilities

Key features

Plug-and-play inference acceleration
Prune redundant visual tokens
FLOPs reduction
HuggingFace LLaVA support
KV Cache compatibility
Attention map visualization

Target Audience

developerdata scientist

Integrations

Not yet documented

Pricing & Plans

Open Source

Free

FAQs

What kind of performance improvements can I expect with FastV?

FastV can achieve up to 45% theoretical FLOPs reduction without harming performance. Latency tests show significant reductions, for example, a 13B model with FastV (K=2 R=75%) can reduce latency from 0.203s to 0.124s per first output token on an A100 GPU.

Which large vision-language models does FastV support?

FastV primarily supports HuggingFace LLaVA models. The project provides specific instructions and demo scripts for integrating FastV with LLaVA-1.5-13b-hf and other compatible LLaVA versions from HuggingFace's transformers library.

How does FastV handle KV Cache for improved efficiency?

FastV is compatible with KV Cache, which can be activated by setting `use_cache=True` during generation. This results in consistent CUDA memory usage reduction and a modest decrease in latency, particularly beneficial for tasks involving many image tokens like video understanding.

Trending

Subcategories trending in Coding & Development

Code Assistants DevOps & Infrastructure No-Code / Low-Code Testing & QA Backend & APIs Prompt Engineering

Trending

Explore

Browse AI tools by category

Content & Design Productivity & Business Coding & Development AI Agents & Automation Research & Education Wellness & Lifestyle Career Development Marketing & Growth Data & Analytics Customer Support & CX Finance E-commerce