Exact match (EM) is an evaluation metric that marks an output correct only when it is identical to a reference answer after normalization. In AI evaluation, it is a fast, deterministic check for outputs that have one correct form, such as codes, identifiers, and extracted fields.
For example, suppose the reference answer is “New York City” and a model produces four outputs: “New York City”, “new york city”, “NYC”, and “New York City, USA”. Strict string comparison passes 1 of 4, a score of 25%. After lowercasing and removing punctuation and articles, 2 of 4 pass, a score of 50%.
Neither number means the model was right half the time. “NYC” is a correct answer that would satisfy a user, but it still scores zero.
Normalization is the cleanup step applied to both strings before comparing them, such as lowercasing and removing punctuation. Over a dataset, the EM score is the fraction of outputs that match. Every partial or alternative correct answer scores zero.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why exact match matters for AI evaluation
Exact match is the strictest and simplest way to compare an output with a reference. It needs no model, has no evaluator variance, and gives the same result every time you run it.
Exact match is useful for:
- Checking closed-form answers: Score short factual answers, codes, identifiers, and extracted fields that have one canonical form.
- Gating machine-readable output: Confirm that a string a downstream system consumes, such as an enum value or a date in a fixed format, is exactly right.
- Tracking regressions: Compare versions of your own system on the same dataset and normalization rules.
- Setting a baseline: Anchor more flexible semantic or model-based scores against a strict, reproducible number.
Exact match measures string identity. It does not measure meaning, usefulness, or factual truth. Where several phrasings are correct, it understates quality and can rank systems in the wrong order.
How exact match is calculated
For each case, normalize the model output and the reference, then check whether they are identical. The score is the share of cases that match.
Formula:
EM = (1 / N) × Σ 1[ normalize(ŷ_i) = normalize(y_i) ] summed over i = 1 to N
Where:
- N is the total number of evaluated cases.
- ŷ_i is the model output for case i.
- y_i is the reference answer for case i.
- normalize is the stated normalization step, applied to both strings.
- 1[ … ] is an indicator that equals 1 when the two normalized strings are identical and 0 otherwise.
The denominator N includes every case. That means empty outputs, refusals, and unparseable responses all count as misses.
If you drop failures from the denominator, EM becomes a score over only the cases the system chose to answer in the right shape. That is a different number, and usually a much more flattering one.
Choosing normalization rules
The normalization rules are part of the metric. If you do not report them, nobody can reproduce or compare your score.
A typical normalization pipeline for short-answer question answering looks like this:
- Lowercase both strings.
- Strip leading and trailing whitespace, and collapse repeated internal whitespace.
- Remove punctuation, or a stated subset of it.
- Remove articles (“a”, “an”, “the”) if your task treats them as non-substantive.
- Apply Unicode normalization so accented and composed characters compare consistently.
Calculate exact match in Python
The following code applies those rules and scores the four outputs from the opening example. It uses only the Python standard library.
import re
import string
import unicodedata
def normalize(text):
text = unicodedata.normalize("NFKC", text)
text = text.lower()
text = "".join(ch for ch in text if ch not in string.punctuation)
text = re.sub(r"b(a|an|the)b", " ", text)
return " ".join(text.split())
def exact_match(outputs, reference, normalizer=None):
prep = normalizer or (lambda s: s)
hits = [prep(out) == prep(reference) for out in outputs]
return hits, sum(hits) / len(hits)
reference = "New York City"
outputs = ["New York City", "new york city", "NYC", "New York City, USA"]
for label, fn in [("raw", None), ("normalized", normalize)]:
hits, score = exact_match(outputs, reference, fn)
marks = ", ".join("pass" if h else "fail" for h in hits)
print(f"{label:<11} {score:.0%} ({marks})")
Output:
raw 25% (pass, fail, fail, fail)
normalized 50% (pass, pass, fail, fail)
Code-based evaluators in Phoenix follow this same pattern of explicit, inspectable rules. For how to wire a check like this into an experiment, see the Phoenix guide to running experiments with code evaluators and the reference on using evaluators in Phoenix experiments.
Worked example: normalization decides the score
Here are the same four outputs, scored against the reference “New York City” under both settings.
| Output | Raw match | After lowercase and punctuation and article removal | Why |
|---|---|---|---|
| “New York City” | pass | pass | Identical |
| “new york city” | fail | pass | Case difference only |
| “NYC” | fail | fail | Correct fact, different string |
| “New York City, USA” | fail | fail | Correct fact, extra text |
The normalization choice alone moved the score from 25% to 50%. The metric is working as designed here. It is just being asked a question it cannot answer.
The fourth output shows a different kind of problem. The model knew the answer, but nobody told it to reply with only the city name. This is an instruction problem, not a knowledge problem.
Exact match treats format compliance and correctness as the same thing. That is exactly what you want for a field that feeds a database. For a human reader, it can be a misleading artifact.
How to inspect and debug exact match results
A low EM score is a starting point for investigation. These checks help you find out what it actually means.
- Read the near misses: Sort failures by edit distance or token overlap with the reference. Clusters usually point to a normalization gap, a missing format instruction, or a reference with several valid forms, and each needs a different fix.
- Audit the references: Some EM failures are reference errors, such as outdated answers, one arbitrary phrasing among many, or inconsistent formatting in the dataset.
- Report the spec and sample size: “EM 0.71 (lowercase, punctuation and articles stripped, n = 500)” is reproducible. “EM 0.71” alone is not.
- Slice the score: EM on short extractive answers and EM on generated explanations behave like different metrics. Report them separately.
For more on keeping references trustworthy, see dataset curation. For more on reporting scores with clear denominators, see evaluation metrics.
Limitations of exact match
The central limitation is that string equality is not semantic correctness. Several other problems follow from that.
- Valid alternatives score zero: Synonyms, different date formats, equivalent units, and harmless extra words all count as misses. This makes EM understate quality on generative tasks.
- Rankings can flip on style: EM penalizes more verbose systems, which are sometimes better. Two systems can swap places purely because one was prompted to answer tersely.
- Right string, wrong reason: A model can match the reference by copying it from retrieved context without answering the actual question. EM cannot tell the difference.
- Multiple references need extra work: Storing several valid answers per case reduces false negatives. It also requires dataset work and a stated rule, usually that a match against any reference counts.
- Normalization does not transfer: Case, punctuation, and whitespace carry meaning in multilingual text, code, and mathematics. Review a pipeline built for English short answers before reusing it there.
Where you need graded rather than binary credit on structured outputs, treat EM as one signal among several. For more, see accuracy.
Exact match vs. semantic and model-based evaluation
Exact match sits at the strict end of a range of evaluation methods.
Token-overlap scores, such as F1 over answer words, forgive extra or missing words. They still reward surface similarity rather than meaning.
Embedding similarity and large language model (LLM) judges can credit “NYC” for “New York City”. They handle paraphrase and can assess reasoning. In exchange, they add evaluator variance, cost, and failure modes of their own.
A practical rule is to match the metric to whoever consumes the answer:
- A downstream system consumes the string: Use EM when any deviation breaks it, such as stock keeping unit (SKU) codes, enum values, SQL fragments, or dates in a fixed format.
- A human consumes the answer: Use a semantic evaluator when several phrasings succeed.
Many pipelines need both. EM or a schema check acts as a format gate, and a judge evaluates substance. For more on that layering, see LLM evaluation. For the underlying question of whether an output is right, see correctness.
Frequently asked questions
Should I normalize case and punctuation before exact match?
Usually yes for natural-language short answers, and you should write the decision down with the score. For code, identifiers, passwords, and case-sensitive codes, do not normalize. Doing so would mark genuinely different outputs as equal.
What is a good exact match score?
There is no portable target number. EM depends on the task’s answer space, the reference style, and the normalization rules. Compare EM across versions of your own system on the same dataset and spec, and treat comparisons across papers or products as unreliable.
How do I handle questions with several correct answers?
Store multiple references and count a case as correct if the output matches any of them after normalization. If the valid answers are open-ended, such as paraphrases, explanations, or summaries, exact match is the wrong primary metric no matter how many references you store.
Why did my exact match score drop after I improved the prompt?
Check the near misses before assuming quality fell. A prompt change that makes answers more complete often adds words that break string identity. If the added text helps users, change the metric rather than the system.
Putting exact match to work
When you report EM, include the normalization rules, the number of cases, and the fact that failures stay in the denominator. Without those, the number cannot be reproduced.
Use EM where the answer space is genuinely closed, and pair it with a semantic evaluator where it is not. Review the near misses after every run, because they show whether the problem is the model, the prompt, or the references.
If you run experiments in Arize Phoenix, you can implement EM as a code evaluator and score every version against the same dataset.
