BLEU is a metric that scores generated text by how much of its word sequence overlaps with one or more human reference texts. It stands for Bilingual Evaluation Understudy, it was introduced by researchers at IBM in a 2002 paper on automatic evaluation of machine translation, and it produces a number between 0 and 1, often reported multiplied by 100. Higher means more overlap with the reference.
That is the entire idea. BLEU counts matching n-grams, penalizes output that is shorter than the reference, and combines the counts into one score. It does not read the text, it does not check whether anything is true, and it has no representation of meaning. Every strength and every failure of the metric comes from that fact.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- BLEU measures n-gram overlap between a candidate and one or more reference texts, combined as a geometric mean of modified precisions and multiplied by a brevity penalty.
- Scores run from 0 to 1, but they are only comparable within an identical setup: same references, same tokenization, same n-gram order, same corpus.
- BLEU is a corpus-level metric. Sentence-level BLEU is noisy and hits zero whenever any n-gram order has no matches, which is why smoothing is required to use it per example.
- BLEU cannot recognize a correct paraphrase and cannot detect a fluent false statement. It rewards surface similarity, not correctness.
- For open-ended generation where no single reference exists, BLEU has nothing to compare against. Criteria-based grading is the workable alternative.
How BLEU is calculated
Two pieces: modified n-gram precision, and a brevity penalty.
Modified n-gram precision. For each n-gram order, count how many of the candidate’s n-grams appear in a reference, clipping each count at the maximum in any single reference.
Brevity penalty. Precision alone rewards short output, so BLEU compensates with a length penalty when the candidate is shorter than the reference.
BP = 1 if c > r
BP = exp(1 - r/c) if c <= rc is the candidate length in tokens and r is the reference length. A short candidate against a longer reference is penalized before precision is considered.
Put together, with N usually set to 4:
BLEU = BP * exp( sum over n = 1..N of w_n * log(p_n) )The exp of a weighted sum of logs is a geometric mean. That detail matters more than it looks: if any single p_n is zero, the whole score collapses to zero.
A worked example
Reference: the cat is on the mat Candidate: the cat sat on the mat
One word differs. Counting clipped matches by n-gram order:
| Order | Candidate n-grams | Clipped matches | Precision |
|---|---|---|---|
| 1 | 6 | 5 | 0.833 |
| 2 | 5 | 3 | 0.600 |
| 3 | 4 | 1 | 0.250 |
| 4 | 3 | 0 | 0.000 |
Both texts are six tokens long, so BP = 1. Standard BLEU-4 takes the geometric mean of all four precisions, and because p_4 is zero, BLEU-4 for this pair is 0. Restrict to BLEU-2 and you get sqrt(0.833 * 0.600) = 0.707, a different verdict on the same sentence.
BLEU is defined over a corpus. Aggregating counts across thousands of segments before dividing keeps any single missing 4-gram from zeroing the result.
Where BLEU breaks on LLM output
BLEU was designed for machine translation, a task with a fairly constrained output space and real reference translations available. Applying it to general LLM output breaks in predictable ways.
It punishes valid paraphrases. Reference: “the cat sat on the mat.” Candidate: “a feline rested on the rug.” The meaning is intact and BLEU is close to zero. Any task where many different wordings are equally correct, which is most tasks, is measured badly by n-gram overlap.
It rewards fluent and wrong. Reference: “the meeting was moved to Friday.” Candidate: “the meeting was not moved to Friday.” Nearly every n-gram matches and the statement is the opposite of the truth.
It needs a reference that often does not exist. There is no gold answer for a support chatbot reply or an agent’s plan.
It does not say what to fix. A BLEU drop tells you overlap fell, not whether the model started hallucinating or changed formatting.
These limits are the reason evaluation for generative systems moved toward graded criteria and model-based judging.
When BLEU still earns its place
BLEU is not worthless, it is narrow. It is fast, deterministic, free to compute, and requires no model call, which makes it a reasonable regression check: run it on a fixed set with fixed references on every build and alert when it moves sharply. That property is what makes it easy to drop into LLM evaluations in CI/CD pipelines next to slower graders. For actual machine translation with real reference translations, it remains a standard reported figure.
The practical pattern is layered. Keep a cheap overlap metric as a tripwire, and use an LLM as a judge for the properties that matter in production: faithfulness to retrieved context, relevance, tone, and refusals. Pick which evaluation metric answers which question rather than expecting one number to cover everything, and attach the results to the spans that produced them, which is what agent evaluation is for.
FAQ
What is a good BLEU score?
There is no universal threshold, and any specific number quoted without context is meaningless. BLEU depends on the language pair, the domain, the tokenizer, the smoothing method, and how many references were used. Adding a second reference translation raises the score without changing the model. Treat BLEU as comparable only against another run of the identical configuration, and report the configuration with the number.
What is the BLEU score formula?
BLEU = BP * exp( sum of w_n * log(p_n) ) over n-gram orders 1 through N, where p_n is the modified precision at order n, w_n is usually 1/N with N = 4, and BP is the brevity penalty exp(1 - r/c) when the candidate is shorter than the reference and 1 otherwise.
What is the difference between BLEU and ROUGE?
BLEU is precision-oriented: of the n-grams the candidate produced, how many appear in the reference. ROUGE is recall-oriented: of the n-grams in the reference, how many the candidate recovered. That makes BLEU the conventional choice for translation, where saying extra things is a fault, and ROUGE the conventional choice for summarization, where coverage of the source matters. Both are surface overlap metrics and both share the same blindness to meaning.
Why is BLEU a poor metric for evaluating chatbots and agents?
Because it requires a reference answer and assumes correctness means matching it. Open-ended responses have many correct forms and usually no gold text at all, so overlap measures writing style rather than quality. Agent behavior compounds the problem: a session is judged on whether the task was completed, not on word choice.
Can BLEU detect hallucinations?
No. A hallucinated sentence built from words that appear in the reference or the retrieved context can score highly, and a correct answer phrased differently can score near zero. Detecting unsupported claims requires comparing the response against its source, which is a faithfulness check rather than an overlap count.
