Metric for Evaluation of Translation with Explicit Ordering (METEOR) score is a metric that measures the quality of generated text based on the alignment between the generated text and the reference text. The metric is based on the harmonic mean of unigram precision and recall, with recall weighted higher than precision. While the main difference between ROUGE and BLEU is that BLEU score is precision-focused and ROUGE score focuses on recall, the METEOR metric on the other hand was designed to fix some of the problems found in the more popular BLEU and ROUGE metrics and also produce good correlation with human judgment at the sentence or segment level. Eval engineers use METEOR when they need an automatic score that rewards lexical variation and ordering without ignoring recall.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- METEOR aligns unigrams between hypothesis and reference, allowing synonym and stem matches beyond exact string overlap.
- The core score uses a recall-weighted F-mean (default recall weight 9 versus precision weight 1), then applies a fragmentation penalty for non-contiguous matches.
- Segment-level evaluation is METEOR’s strength; corpus-level aggregation behaves differently from BLEU n-gram counting.
- Synonym resources and language-specific stemmers affect scores; lock toolchain versions for reproducible evals.
- METEOR complements BLEU and ROUGE; no single n-gram metric captures fluency, adequacy, and factual grounding alone.
How METEOR computes a score
METEOR proceeds in stages:
- Alignment. Build a mapping between unigrams in the candidate translation (hypothesis) and the reference. Exact matches take priority, then stem matches (e.g., run/running), then synonym matches from a lexical resource such as WordNet for supported languages.
- Precision and recall on unigrams. Precision is matched unigrams in the hypothesis divided by total hypothesis unigrams. Recall is matched unigrams divided by total reference unigrams.
- Weighted harmonic mean. METEOR uses an F-mean with weights that favor recall (commonly alpha = 0.9 on recall):
F_mean = (P * R) / (alpha * P + (1 – alpha) * R)
- Fragmentation penalty. Identify matched chunks that appear in the same order in both strings. The penalty increases when matches scatter into many small fragments, discouraging correct words in jumbled order.
- Final score. Multiply the F-mean by (1 – penalty), producing a value often reported on a 0 to 1 scale (some implementations scale to 0 to 100).
The explicit ordering piece in the name refers to how chunk continuity influences the penalty, not to a full syntactic parse.
Comparison with BLEU and ROUGE
BLEU emphasizes n-gram precision with a brevity penalty. It correlates with corpus-level translation quality but can underrate hypotheses that use valid synonyms. ROUGE variants, common in summarization, stress recall of reference n-grams. METEOR sits between those philosophies at unigram level with softer matching and a recall tilt.
None of these metrics detect hallucinated facts or toxic outputs. They measure lexical overlap under alignment rules, not truth. Pair METEOR with groundedness checks for RAG systems and task-specific rubrics for high-risk domains.
When to use METEOR
Machine translation. METEOR was introduced for translation evaluation and remains useful for comparing systems on segment-level test sets.
Text generation benchmarks. Summarization and dialogue teams sometimes report METEOR alongside ROUGE when human ratings are sparse.
Regression testing. When a prompt or decoding change alters wording but preserves meaning, METEOR often moves less harshly than BLEU pure precision.
Avoid treating small METEOR deltas as product wins without human review on representative samples. Automatic metrics compress multidimensional quality into one scalar.
Implementation and reproducibility
METEOR depends on external linguistic resources for synonym matching. Pin package versions, language IDs, and tokenizer behavior. Hypothesis and reference preprocessing (lower casing, punctuation stripping) must match across experiments.
For multilingual work, confirm whether your METEOR port supports the target language or falls back to exact matching only. Unsupported languages reduce METEOR toward simpler overlap scores.
Compute METEOR per sentence, then macro-average (mean of sentence scores) when you care about segment quality. Micro-averaging pooled counts behaves closer to corpus BLEU. Document which aggregation you report.
Limits and failure modes
Reference multiplicity. One reference under-rewards valid paraphrases; multiple references help all n-gram metrics, including METEOR.
Length mismatch. Very short hypotheses can game precision while failing adequacy; read side-by-side outputs when scores disagree with intuition.
Domain mismatch. Synonym nets built for news text may misalign technical terms; domain terms may need custom dictionaries.
No semantic entailment. Correct answers with no lexical overlap still score near zero.
Fitting METEOR into eval harnesses
Treat METEOR as one column in an eval matrix: exact match for structured tasks, BLEU or chrF for translation sweeps, LLM judges for rubric dimensions METEOR cannot see. Evaluation harness design describes how to wire multiple scorers into CI and batch pipelines without letting any single metric block releases uncritically.
Platform guides on LLM and agent evaluation platforms cover storing eval runs, comparing model versions, and attaching human labels when automatic scores conflict.
For teams running evals in CI, writing LLM evals as tests shows how to set thresholds that fail builds on regressions while keeping flaky metrics from blocking every deploy.
FAQ
What is a good METEOR score?
It depends on language, task, and reference count. Compare relative improvements on the same test set rather than absolute numbers from papers. Report variance across sentences, not only means.
Does METEOR work for summarization?
Yes, though ROUGE remains more common. METEOR can reward synonym overlap in abstractive summaries when references align. Human evaluation still matters for coherence and factuality.
Why is recall weighted higher?
Translation adequacy often requires covering reference content; missing words hurt comprehension more than extra function words hurt precision. The 9:1 recall weight reflects that design choice in the default formula.
Can I use METEOR for chatbots?
You can when turns have reference replies (gold support scripts). Open-ended dialogue rarely has single references; METEOR is a poor primary metric there.
How does METEOR handle word order?
Through the fragmentation penalty on matched chunks. Permutations that preserve contiguous phrases score better than the same words scattered randomly, but METEOR is not a full tree-based evaluation like some parse metrics.
