Sumeval

Visit Tool

sumeval is a well-tested, multi-language evaluation framework for text summarization. It provides ROUGE-X and BLEU scores, supporting English, Japanese, and Chinese.

Claim this tool

No Views Yet

At a glance

Pricing

Open Source

Free tier

Yes

API

Yes

Skill level

Technical

About

What is sumeval?

sumeval is an open-source, multi-language evaluation framework designed for text summarization. It allows users to test and compare various text summarization algorithms with high accuracy. The framework is thoroughly tested, with ROUGE-X scores validated against the original Perl script (ROUGE-1.5.5.pl) and BLEU scores matching the official mteval-v13a.pl script via SacréBLEU. Beyond English, sumeval supports Japanese and Chinese, with an extensible architecture for adding other languages. It is implemented purely in Python and can be used programmatically or via the command line, making it a versatile tool for researchers and developers in natural language processing.

Best used for

Ideal for researchers and students who need to accurately evaluate text summarization models, compare different algorithms, and validate their performance. Especially valuable for those working with multi-language summarization, including English, Japanese, and Chinese.

Common actions

evaluate summarization models

compare text summarizers

calculate ROUGE scores

calculate BLEU scores

open-sourceworkflowscollaboration"AI Agents"github copilotautomated workflowlow-code/no-codedeepfakeface swapping

Capabilities

Key features

ROUGE-X score calculation
BLEU score calculation
Multi-language support
Pure Python implementation
Command-line interface
Tested against official scripts

Target Audience

professorstudent

Integrations

Not yet documented

Pricing & Plans

Open Source

Free

FAQs

What languages does sumeval support for text summarization evaluation?

sumeval natively supports English, Japanese, and Chinese for text summarization evaluation. Its architecture is designed to be easily extensible, allowing developers to add support for other languages as needed by implementing custom language classes.

How accurate are the ROUGE and BLEU scores calculated by sumeval?

The ROUGE-X scores in sumeval are rigorously tested and compared against the original Perl script (ROUGE-1.5.5.pl) for accuracy. The BLEU score calculation uses SacréBLEU, which produces values consistent with the official mteval-v13a.pl script used by WMT.

What are the main dependencies for using sumeval?

BLEU scores depend on SacréBLEU. For ROUGE-BE, spaCy is required. Japanese language support needs janome or MeCab, and GiNZA for ROUGE-BE. Chinese support requires jieba, and pyhanlp for ROUGE-BE.

Trending

Subcategories trending in Research & Education

Academic Research Study Assistants Knowledge Management Course Creation Scientific Computing Language Learning

Trending

Also listed in

This tool also appears in

Coding & Development › Open Source & Models

Explore

Browse AI tools by category

Content & Design Productivity & Business Coding & Development AI Agents & Automation Research & Education Wellness & Lifestyle Career Development Marketing & Growth Data & Analytics Customer Support & CX Finance E-commerce