BLEU Score
BLEU (Bilingual Evaluation Understudy) is a precision-based metric that scores machine-generated text, most commonly machine translation output, by comparing overlapping word sequences (n-grams) against one or more human-written reference…
Definition
BLEU (Bilingual Evaluation Understudy) is a precision-based metric that scores machine-generated text, most commonly machine translation output, by comparing overlapping word sequences (n-grams) against one or more human-written reference texts.
Overview
BLEU was introduced in a 2002 IBM research paper as a fast, automatic alternative to costly human evaluation of machine translation quality. It works by computing modified n-gram precision — the fraction of n-grams (contiguous sequences of n words, typically for n from 1 to 4) in the generated output that also appear in the reference text or texts, with a cap to prevent a model from gaming the score by repeating a common word excessively. These precision scores across different n-gram lengths are combined, typically via a geometric mean, and multiplied by a brevity penalty that discourages the model from producing suspiciously short outputs that would otherwise score well by only including safe, high-precision words. BLEU scores range from 0 to 1 (often reported as 0 to 100), where higher scores indicate closer overlap with the reference translations. Because BLEU only measures surface-level n-gram overlap, it does not directly assess grammaticality, meaning preservation, or fluency, and it can penalize valid translations or generations that use different but equally correct wording than the reference. Despite its limitations, BLEU remained the dominant automatic metric for machine translation research for nearly two decades due to its speed, reproducibility, and reasonable correlation with human judgment at the corpus level, though its correlation is weaker for individual sentences. It is typically less suited to open-ended generation tasks like summarization or dialogue, where ROUGE or embedding-based and LLM-judge metrics are often preferred. In recent years, learned metrics like BERTScore and LLM-based evaluation have gained ground for capturing semantic similarity that pure n-gram overlap misses, but BLEU remains a standard baseline reported in most machine translation papers.
Key Concepts
- Measures n-gram precision overlap between generated text and reference translations
- Combines precision scores across n-gram lengths (typically 1 to 4) via geometric mean
- Applies a brevity penalty to discourage overly short outputs
- Scores range from 0 to 1, commonly reported as 0 to 100
- Supports multiple reference translations per source sentence
- Correlates reasonably well with human judgment at the corpus level
- Does not directly assess grammaticality or semantic meaning preservation
- Introduced by IBM researchers in 2002 for machine translation evaluation