What Is BERTScore?

BERTScore is a metric that scores generated text by comparing its tokens with a reference text’s tokens using contextual embeddings. In AI evaluation, it gives credit for valid rewording that word-overlap metrics miss, which makes it useful for screening summaries, translations, and paraphrases.

For example, suppose the reference is “The patient was discharged on Tuesday.” Candidate A says “The patient left hospital on Tuesday.” It shares few exact words with the reference, so a word-overlap metric penalizes it. BERTScore matches “left hospital” closely to “was discharged” and scores A well.

Candidate B says “The patient was discharged on Thursday.” It differs by one word, so it also scores well on token similarity. Yet it states the wrong day.

That is the main thing to understand about BERTScore. It measures similarity to a reference, not truth, and not agreement with a source document.

A token is a word or word piece. A contextual embedding is a vector that represents a token in light of the words around it. BERT stands for Bidirectional Encoder Representations from Transformers, the pretrained model family the metric was first built on.

Try Arize AX

Build better agents with Arize

Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.

Get started View AX docs

Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.

Why BERTScore matters for AI evaluation

Many generation tasks have more than one good answer. Metrics that count exact word overlap punish valid rewording, while BERTScore relaxes the match to contextual similarity.

BERTScore is useful for:

• Screening paraphrase-heavy tasks: Score summaries, translations, and rewrites where many phrasings are acceptable.
• Regression testing: Track whether a prompt or model change moves outputs closer to or further from trusted references.
• Comparing against overlap metrics: See when a low word-overlap score reflects valid rewording rather than a bad output.
• Building an evaluator mix: Pair a similarity signal with checks for factual support and task correctness.

BERTScore needs a reference for every candidate. Without a reference, there is no score.

How BERTScore works

BERTScore was introduced by Zhang, Kishore, Wu, Weinberger, and Artzi in BERTScore: Evaluating Text Generation with BERT (2019). The calculation has three steps.

1. Embed each token: A pretrained model such as BERT turns every token in the reference and the candidate into a contextual embedding.
2. Match tokens greedily: Each candidate token is matched to its most similar reference token by cosine similarity, and each reference token is matched to its most similar candidate token.
3. Average the matches: The average best-match similarity gives precision and recall, which combine into an F1 score.

Precision asks how much of the candidate is covered by the reference. Recall asks how much of the reference is covered by the candidate. F1 is the harmonic mean of the two.

For background on the vectors being matched, see embeddings. For more on the similarity measure, see cosine similarity.

How to calculate BERTScore

Suppose the reference has tokens x1 through xk and the candidate has tokens x̂1 through x̂m.

Formula:

P_BERT = (1 / |x̂|) × Σ over each x̂_j in x̂ of [ max over x_i in x of (x_i · x̂_j) ]
R_BERT = (1 / |x|) × Σ over each x_i in x of [ max over x̂_j in x̂ of (x_i · x̂_j) ]
F_BERT = (2 × P_BERT × R_BERT) / (P_BERT + R_BERT)

Where:

• x is the reference and x̂ is the candidate.
• x_i is the contextual embedding of the i-th reference token.
• x̂_j is the contextual embedding of the j-th candidate token.
• |x| and |x̂| are the number of tokens in the reference and the candidate.
• x_i · x̂_j is the inner product of the two embeddings.
• P_BERT, R_BERT, and F_BERT are precision, recall, and F1.

The embeddings are L2-normalized, which means each vector is scaled to length 1. As a result, the inner product equals the cosine similarity.

Optional settings that change the score

The original paper also describes two optional settings:

• Inverse document frequency (IDF) weighting: Gives rare tokens more weight than common ones.
• Baseline rescaling: Stretches the compressed range of raw scores so differences are easier to see.

Both change the reported values. A BERTScore number without its configuration cannot be reproduced.

Calculate BERTScore in Python

The following code implements the precision, recall, and F1 formulas with NumPy. It uses hand-picked 3-dimensional vectors instead of real embeddings, so you can see the greedy matching without downloading a model.

The vectors are hypothetical. Real contextual embeddings have many more dimensions, and real raw scores sit in a narrower band.

import numpy as np

def bertscore(ref_vecs, cand_vecs):
    ref = np.array(ref_vecs, dtype=float)
    cand = np.array(cand_vecs, dtype=float)
    # L2-normalize so the inner product equals cosine similarity.
    ref /= np.linalg.norm(ref, axis=1, keepdims=True)
    cand /= np.linalg.norm(cand, axis=1, keepdims=True)
    sim = ref @ cand.T  # rows: reference tokens, columns: candidate tokens
    precision = sim.max(axis=0).mean()  # best match for each candidate token
    recall = sim.max(axis=1).mean()     # best match for each reference token
    f1 = 2 * precision * recall / (precision + recall)
    return precision, recall, f1

# Hypothetical 3-dimensional token vectors, chosen by hand for illustration.
vec = {
    "patient":    [1.0, 0.0, 0.0],
    "discharged": [0.0, 1.0, 0.2],
    "left":       [0.2, 0.8, 0.1],
    "hospital":   [0.5, 0.7, 0.0],
    "tuesday":    [0.0, 0.1, 1.0],
    "thursday":   [0.0, 0.2, 0.95],
    "rain":       [0.4, -0.3, -0.5],
    "expected":   [-0.3, 0.4, -0.6],
}

reference = ["patient", "discharged", "tuesday"]
candidates = {
    "A (paraphrase)": ["patient", "left", "hospital", "tuesday"],
    "B (wrong day)": ["patient", "discharged", "thursday"],
    "C (off-topic)": ["rain", "expected"],
}

for name, tokens in candidates.items():
    p, r, f = bertscore([vec[t] for t in reference], [vec[t] for t in tokens])
    print(f"{name:<15} P={p:.3f}  R={r:.3f}  F1={f:.3f}")

Output:

A (paraphrase)  P=0.941  R=0.989  F1=0.965
B (wrong day)   P=0.998  R=0.998  F1=0.998
C (off-topic)   P=0.459  R=0.068  F1=0.118

In this toy setup, the paraphrase scores well and the off-topic candidate scores low. The wrong-day candidate scores highest of all, because “Thursday” sits close to “Tuesday” in the vector space.

For real scores, use an established BERTScore implementation with a pretrained model, and record its configuration.

Worked example: what BERTScore catches and misses

Return to the reference “The patient was discharged on Tuesday.” The following summary describes the pattern you can expect. It is illustrative, not a set of measured scores.

Illustrative only (not measured scores):
A (true paraphrase):  high BERTScore, low exact-overlap  -> BERTScore helps
B (one wrong fact):   high BERTScore, high exact-overlap -> neither catches it
C (fluent, off-topic): lower BERTScore                     -> BERTScore helps

Candidate A is where BERTScore beats an n-gram metric, which counts matching word sequences. The BLEU score (Bilingual Evaluation Understudy) is a common example. Candidate C is where BERTScore correctly gives a lower score to fluent text that is off-topic.

Candidate B is the central caution. Embedding similarity rewards fluent, on-topic text, including text that is wrong.

To check whether claims are supported by a source, you need a different kind of metric. For more, see faithfulness.

How to measure and debug BERTScore

A BERTScore number is only meaningful on a pinned setup. Before you rely on it, work through these steps.

• Pin the configuration: Record the embedding model and layer, whether IDF weighting is on and which corpus it was computed on, baseline rescaling, language, and library version.
• Use trusted references: Compute scores on a fixed evaluation dataset whose references you have reviewed.
• Sanity-check the ordering: Score known-good paraphrases, known-wrong factual edits, and empty or copied candidates. Confirm the ranking matches your own judgment.
• Compare with human review: Check metric rankings against a sample of human evaluation on your own task. Agreement reported on other tasks does not transfer automatically.

Treat BERTScore as one candidate evaluator whose agreement with reviewers you validate. For more on choosing and validating metrics, see evaluation metrics.

BERTScore vs. exact-overlap metrics

BLEU and exact match reward surface overlap and punish valid rewording. BERTScore relaxes the match to contextual similarity. That suits summarization and translation screening, where many phrasings are acceptable.

Neither family checks facts or follows instructions. Neither knows whether required content is missing, beyond what token recall captures against that one reference.

A single reference also understates the range of good answers, so recall against it has a ceiling. For open-ended generation, teams often pair a similarity metric with a rubric-based judge or task-specific checks rather than picking one winner.

Limitations of BERTScore

BERTScore is a useful similarity signal with several known blind spots.

• Similarity is not truth: Negations, swapped entities, and wrong numbers can sit close together in embedding space.
• Scores are compressed: Raw scores fall into a narrow high band, so small differences are often noise.
• Model bias carries over: The metric inherits the biases of its embedding model. It behaves worse for languages and domains that model handles poorly.
• References cap the score: A wrong or narrow reference makes good candidates score badly.

Use BERTScore as a screening and regression signal on a pinned setup. Keep a human-reviewed sample in the loop for any release decision.

Frequently asked questions

Does a high BERTScore mean the answer is correct?

No. A high BERTScore means the answer is contextually similar to the reference, token by token. A fluent answer with a wrong date, name, or negation can still score highly.

Do I need a reference text for BERTScore?

Yes. BERTScore is reference-based, so every candidate is matched against at least one reference. Without references, use a different evaluation design.

Why do BERTScore values differ between papers?

Different embedding models and layers, IDF weighting, baseline rescaling, tokenization, and language all change the values. Only compare numbers produced with the same configuration.

When should I use BERTScore?

Use it for screening and regression testing on tasks with trusted references and many acceptable phrasings, such as summarization, translation, and paraphrase. Run it alongside a check that targets factual correctness.

Putting BERTScore to work

When you report BERTScore, record the embedding model, layer, IDF setting, baseline rescaling, language, and library version. Without them, nobody can reproduce or compare the number.

Validate the metric on your own task before trusting it. Score known-good and known-wrong examples, and compare the ranking with human review.

Pair BERTScore with an evaluator that checks factual support. For example, you can run a similarity score and an LLM evaluation on the same dataset in Arize Phoenix and review the cases where they disagree.