BM25 is a ranking function that scores how relevant a document is to a search query based on the query words it contains. In AI engineering, BM25 is a common retrieval method in search and retrieval-augmented generation (RAG) systems, especially when exact names, codes, or rare terms matter.
For example, suppose a user searches your documentation for “refund webhook timeout.” A short troubleshooting chunk that mentions “webhook” and “timeout” twice each scores about 16.2. A longer architecture page that mentions “webhook” and “refund” once each scores about 7.8, so the troubleshooting chunk ranks first.
BM25 stands for Best Matching 25. It is often called Okapi BM25.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why BM25 matters for AI retrieval
Many AI applications depend on retrieving the right text before a model can answer. BM25 is a fast, explainable way to find passages that contain the exact words a user typed.
BM25 is useful for:
- Finding exact identifiers: Retrieve passages that contain product names, error codes, function names, or rare entities.
- Building hybrid retrieval: Pair keyword scores with vector search so you catch both exact matches and paraphrases.
- Debugging retrieval misses: See which query terms contributed to a score, term by term.
- Setting a baseline: Compare a new embedding model or reranker against a simple, well-understood ranking method.
Embedding-based retrieval can miss exact strings that matter to the user. BM25 often fills that gap.
For more on how retrieval fits into a RAG pipeline, see retrieval-augmented generation (RAG).
How BM25 works
BM25 is lexical. It matches words, not meanings. Each query term adds to a document’s score based on three ideas.
Term frequency: A document that contains a query term more often scores higher. The gain shrinks with each repeat, so the tenth occurrence adds much less than the first.
Inverse document frequency (IDF): A term that appears in few documents is more informative than a term that appears in most of them. Rare terms get a high weight and common words get a weight near zero.
Length normalization: A long document contains more words, so it is more likely to match a term by chance. BM25 adjusts for this, so a long document does not win only because it has more text.
The final score is the sum of the contributions from each query term. Terms that do not appear in the document contribute nothing.
Term frequency saturation
Consider a document of average length with the common setting k1 = 1.2. The term frequency part of the formula grows like this:
occurrences term frequency factor
1 1.000
2 1.375
3 1.571
10 1.964
The factor approaches a ceiling of k1 + 1, which is 2.2 here. This means stuffing a document with a keyword quickly stops helping it rank.
Inverse document frequency
Suppose your collection has 10,000 chunks. If “webhook” appears in 40 of them, its IDF is about 5.51. If “the” appears in 9,000 of them, its IDF is about 0.11. A match on “webhook” therefore counts roughly 50 times more than a match on “the.”
How to calculate a BM25 score
For a query Q with terms q1 through qn, BM25 adds up a score for each term in document D.
Formula:
score(D, Q) = Σ (i = 1 to n) IDF(qi) × [ f(qi, D) × (k1 + 1) ] / [ f(qi, D) + k1 × (1 − b + b × |D| / avgdl) ]
Where:
- f(qi, D) is how often term qi occurs in document D.
- |D| is the length of document D, usually in words or tokens.
- avgdl is the average document length in the collection.
- k1 controls term frequency saturation. It is commonly set between about 1.2 and 2.0.
- b controls how strongly document length is normalized. It is commonly set to 0.75.
A widely used IDF form follows the Stanford Introduction to Information Retrieval treatment of Okapi BM25.
Formula:
IDF(qi) = ln( 1 + (N − n(qi) + 0.5) / (n(qi) + 0.5) )
Where:
- N is the number of documents in the collection.
- n(qi) is the number of documents that contain term qi.
- ln is the natural logarithm.
Implementations vary in the exact IDF variant and default parameters. Compare BM25 results across systems only when the variant and parameters are stated.
Calculate BM25 in Python
The following function scores documents with the formulas above, using only the standard library. It is an illustration for learning, not a production search engine.
from math import log
N = 10_000
avgdl = 200
k1, b = 1.2, 0.75
doc_freq = {"refund": 200, "webhook": 40, "timeout": 100}
def idf(term):
n = doc_freq[term]
return log(1 + (N - n + 0.5) / (n + 0.5))
def bm25(term_counts, doc_len, query):
score = 0.0
for term in query:
f = term_counts.get(term, 0)
norm = k1 * (1 - b + b * doc_len / avgdl)
score += idf(term) * f * (k1 + 1) / (f + norm)
return score
query = ["refund", "webhook", "timeout"]
docs = {
"A (100 words)": ({"webhook": 2, "timeout": 2}, 100),
"B (300 words)": ({"webhook": 1, "refund": 1}, 300),
"C (100 words)": ({"refund": 3}, 100),
}
for term in query:
print(f"IDF({term}) = {idf(term):.2f}")
for name, (counts, length) in docs.items():
print(f"doc {name}: {bm25(counts, length, query):.2f}")
Output:
IDF(refund) = 3.91
IDF(webhook) = 5.51
IDF(timeout) = 4.60
doc A (100 words): 16.18
doc B (300 words): 7.82
doc C (100 words): 6.88
Worked example: ranking three documentation chunks
Suppose a docs search for “refund webhook timeout” runs over 10,000 chunks with an average length of 200 words. The parameters are k1 = 1.2 and b = 0.75.
| Chunk | Length | Matching terms | BM25 score | Rank |
|---|---|---|---|---|
| A: troubleshooting note | 100 words | webhook x2, timeout x2 | 16.18 | 1 |
| B: architecture page | 300 words | webhook x1, refund x1 | 7.82 | 2 |
| C: billing FAQ | 100 words | refund x3 | 6.88 | 3 |
Chunk A wins because it matches the two rarest query terms, and it matches each of them twice.
Chunk B matches two terms, but its length pulls its score down. The same two matches in a 100-word chunk would score about 11.84.
Chunk C repeats “refund” three times, but saturation limits the gain. Three mentions score 6.88 compared with 4.91 for one mention. It also misses the rarer terms entirely.
This behavior is why BM25 remains useful beside embeddings and vector search. A vector search might favor a page about payment failures in general. BM25 rewards chunks that contain the exact query terms.
How to evaluate and debug BM25 retrieval
Start by checking whether the right document is in the candidate set at all. Then check whether BM25 ranks it near the top.
These metrics help:
- Judging the top results: Precision@K measures how many of the top K results are relevant.
- Finding one known answer: Mean reciprocal rank rewards ranking the single correct document early.
- Separating retrieval from generation: RAG evaluation tells you whether a bad answer came from a retrieval miss or a generation error.
When BM25 misses a document, inspect tokenization, synonyms, and field weighting before changing k1 or b. A term that is never indexed cannot be scored.
For more on judging retrieved results, see retrieval quality.
Limitations of BM25
BM25 is simple and fast, but it has known blind spots.
- No synonym or paraphrase handling: A query for “payment errors” does not match a passage that only says “billing failures.”
- Sensitive to tokenization and spelling: A typo or a different word form can drop a match entirely.
- Terms are scored independently: Word order and negation carry little signal in the standard formula.
- Collection-dependent scores: The same document scores differently after the index changes, because IDF and average length change.
- Parameters need evidence: Tune k1 and b on judged queries, not by feel.
BM25 vs. vector search
BM25 matches words. Semantic search with embeddings matches meaning.
Semantic search can retrieve a passage about “billing failures” for a query about “payment errors,” even with no shared words. BM25 cannot do that.
BM25 is exact and explainable term by term. It is strong on identifiers, error codes, and rare exact terms, where embeddings are often weaker.
Many systems therefore combine both. The two score scales are not directly comparable, so fusing them needs care. For more on combining them, see hybrid search and reciprocal rank fusion.
Frequently asked questions
Why is BM25 still used with embeddings?
Exact terms still matter. Product names, codes, and rare entities are often retrieved more reliably by keyword matching, while embeddings handle paraphrase. Combining the two covers both kinds of failure.
What do k1 and b do in BM25?
k1 controls how quickly repeated terms stop adding to the score. b controls how strongly document length is normalized. The defaults are a starting point, and implementations differ.
Is a higher BM25 score a probability of relevance?
No. A BM25 score is an unbounded ranking score. It is useful for ordering results within one index and parameter setup. It is not a calibrated probability, and it is not comparable across indexes.
Does BM25 understand word order?
Not in its standard form. BM25 scores which terms occur and how often. Any phrase or proximity matching comes from extra features in a particular search engine.
Why does BM25 give common words almost no weight?
Common words appear in most documents, so their IDF is close to zero. In a 10,000-chunk collection, a word found in 9,000 chunks has an IDF of about 0.11.
Putting BM25 to work
When you add BM25 to a retrieval system, record the IDF variant, the values of k1 and b, and how text is tokenized. Without those details, scores cannot be compared or reproduced.
Build a small set of judged queries that includes exact identifiers and paraphrased questions. Measure BM25, vector search, and a hybrid on the same set before choosing.
In a RAG application, trace each retrieval step so you can see which chunks were returned and how they ranked. Tools such as Arize Phoenix can capture those retrieval spans and help you trace a bad answer back to a retrieval miss.
