Nlg-Eval

Visit Tool

nlg-eval is an open-source Python library for evaluating Natural Language Generation (NLG) models. It computes various unsupervised automated metrics like BLEU, METEOR, ROUGE, CIDEr, and SPICE from hypothesis and reference files.

Claim this tool

1View

At a glance

Pricing

Open Source

Free tier

Yes

API

Yes

Skill level

Technical

About

What is nlg-eval?

nlg-eval is a comprehensive open-source Python library designed for the evaluation of Natural Language Generation (NLG) systems. It provides a suite of unsupervised automated metrics, including BLEU, METEOR, ROUGE, CIDEr, SPICE, SkipThought cosine similarity, Embedding Average cosine similarity, Vector Extrema cosine similarity, and Greedy Matching score. The tool takes a hypothesis file and one or more reference files as input, where corresponding rows represent the same example, and outputs the calculated metric values. It offers both functional and object-oriented APIs for evaluating entire corpora or individual sentences, making it flexible for various research and development needs in NLG.

Best used for

Ideal for developers and data scientists who need to evaluate the quality of natural language generation models, compare different model architectures, and track performance improvements. Especially valuable for researchers and practitioners in NLP working on tasks like dialogue systems, summarization, and machine translation.

Common actions

evaluate language models

measure NLG performance

compare text generation

calculate NLP metrics

github copilot"AI Agents"workflowslow-code/no-codecollaborationdeepfakeface swappingopen-sourceautomated workflow

Capabilities

Key features

BLEU score calculation
METEOR score calculation
ROUGE score calculation
CIDEr score calculation
SPICE score calculation
SkipThought similarity
Embedding Average similarity

Target Audience

developerdata scientistprofessorstudent

Integrations

Not yet documented

Pricing & Plans

Open Source

Free

FAQs

What metrics does nlg-eval support for Natural Language Generation?

nlg-eval supports a wide range of unsupervised automated metrics including BLEU, METEOR, ROUGE, CIDEr, SPICE, SkipThought cosine similarity, Embedding Average cosine similarity, Vector Extrema cosine similarity, and Greedy Matching score for comprehensive NLG evaluation.

How do I install nlg-eval and its dependencies?

To install nlg-eval, you first need Java 1.8.0 or higher. Then, install Python dependencies via `pip install git+https://github.com/Maluuba/nlg-eval.git@master`. Finally, run `nlg-eval --setup` to download necessary models and data.

Can nlg-eval be used to evaluate individual sentences or only entire corpora?

nlg-eval offers both functional and object-oriented APIs that allow for the evaluation of both entire corpora (multiple sentences) and individual sentences, providing flexibility for different use cases in NLG assessment.

Trending

Subcategories trending in AI Agents & Automation

AI Frameworks & Infra General-Purpose Agents Workflow Agents Personal Assistants RAG & Document AI Voice Agents

Trending

Explore

Browse AI tools by category

Content & Design Productivity & Business Coding & Development AI Agents & Automation Research & Education Wellness & Lifestyle Career Development Marketing & Growth Data & Analytics Customer Support & CX Finance E-commerce