VLMEvalKit

Visit Tool

VLMEvalKit is a Coding & Development tool that provides an open-source evaluation toolkit for large multi-modality models (LMMs). It supports over 220 LMMs and 80+ benchmarks, enabling one-command evaluation.

Claim this tool

No Views Yet

At a glance

Pricing

Open Source

Free tier

Yes

API

Skill level

Technical

About

What is VLMEvalKit?

VLMEvalKit is an open-source evaluation toolkit designed for large vision-language models (LVLMs), supporting over 220 LMMs and 80+ benchmarks. It simplifies the evaluation process by allowing one-command evaluation without extensive data preparation across multiple repositories. The toolkit uses generation-based evaluation for all LVLMs, offering results with both exact matching and LLM-based answer extraction. Recent updates include improved handling for models with thinking mode and long responses, as well as multi-node distributed inference support for faster evaluations. It aims to provide an easy-to-use, reproducible evaluation environment for researchers and developers.

Best used for

Ideal for developers and professors who need to evaluate the performance of large vision-language models, benchmark new models against existing ones, and reproduce evaluation results efficiently. Especially valuable for researchers looking for an open-source, easy-to-use toolkit that handles data preparation and offers flexible evaluation methods.

Common actions

evaluate AI models

benchmark AI performance

compare vision-language models

reproduce evaluation results

develop custom benchmarks

low-code/no-codeopen-sourcedeepfakecollaborationworkflowsautomated workflowface swapping"AI Agents"github copilot

Capabilities

Key features

Evaluate 220+ LMMs
Support 80+ benchmarks
One-command evaluation
Generation-based evaluation
LLM-based answer extraction
Multi-node distributed inference
Customizable thinking mode parsing

Target Audience

developerprofessor

Integrations

Not yet documented

Pricing & Plans

Open Source

Free

FAQs

What types of models does VLMEvalKit support?

VLMEvalKit supports a wide range of large vision-language models (LVLMs), including over 220 different models. This includes commercial APIs, open-source models, and various series like InternVL, QwenVL, LLaMA, Gemini, and many others, making it a comprehensive evaluation platform.

How does VLMEvalKit handle long model responses during evaluation?

VLMEvalKit has improved support for models generating long responses, especially those exceeding 16k or 32k tokens. It can save prediction files in TSV format, which is recommended over .xlsx to prevent data truncation due to cell character limits.

Can VLMEvalKit speed up evaluations for large-scale models?

Yes, VLMEvalKit supports multi-node distributed inference using tools like LMDeploy or VLLM. This feature can be activated in custom model configurations and significantly speeds up evaluation workflows for large-scale or 'thinking' models, such as InternVL Series, QwenVL Series, and LLaMa4.

Trending

Subcategories trending in Coding & Development

Open Source & Models Code Assistants DevOps & Infrastructure No-Code / Low-Code Backend & APIs Prompt Engineering

Trending

Also listed in

This tool also appears in

Research & Education › Academic Research

Explore

Browse AI tools by category

Content & Design Productivity & Business Coding & Development AI Agents & Automation Research & Education Wellness & Lifestyle Career Development Marketing & Growth Data & Analytics Customer Support & CX Finance E-commerce