Malhar Ujawane

Blog

Explainer·(updated October 31, 2026)

What is BM25? The keyword ranking function, explained

BM25 ranks documents by how often query terms appear, how rare they are and how long the document is. The formula, k1 and b, with a from-scratch Python demo.

What is BM25? The keyword ranking function, explained

BM25 (Okapi BM25) is a keyword ranking function: for each query word it adds up a score based on how often the word appears in a document, how rare the word is across the whole collection, and how long the document is compared with the average. It is the default ranking in Lucene and Elasticsearch, and it is still the keyword half of almost every hybrid RAG system.

It comes out of the probabilistic relevance framework that Stephen Robertson, Karen Sparck Jones and colleagues developed from the 1970s on, and was first used in the Okapi retrieval system, whose team reported it at the early TREC conferences. The best single reference is Robertson and Zaragoza’s 2009 monograph, The Probabilistic Relevance Framework: BM25 and Beyond, which calls it “one of the most successful text-retrieval algorithms”.

How does BM25 work?

For a query with terms q1 to qn and a document d, BM25 is a sum over the query terms:

score(d, q) = sum over terms t in q of  IDF(t) * tf * (k1 + 1) / (tf + k1 * (1 - b + b * dl / avgdl))
  • tf is how many times term t appears in d.
  • dl is the length of d, avgdl the average length across the collection.
  • IDF(t) rewards rare terms. Lucene computes it as log(1 + (N - n + 0.5) / (n + 0.5)), where N is the number of documents and n the number that contain t, per the BM25Similarity javadoc. The 1 + inside the log keeps it from going negative for very common words.
  • k1 and b are the two knobs.

Robertson and Zaragoza write the term weight as equation 3.15 of the monograph and say the full document score “is obtained by summing these term-weights over the… set of query terms”. Everything interesting is in the two knobs.

What do k1 and b do?

k1 controls saturation. Plain term frequency grows without limit: a page that says “refund” 50 times scores 50 times higher than one that says it once. BM25’s tf part, tf * (k1 + 1) / (tf + k1) for a document of average length, approaches a ceiling of k1 + 1. The first mention matters a lot, the tenth hardly at all. Robertson and Zaragoza call this saturation and note that “any one term’s contribution to the document score cannot exceed a saturation point”. Low k1 saturates fast; a very high k1 behaves like raw counts.

b controls length normalization. A long document mentions everything more often simply because it is long. b sets how hard to discount for that, from 0 (ignore length) to 1 (fully normalize by length relative to the average). The monograph explains why you want a middle value: some documents are long because the author is verbose, others because they cover more ground, and a real collection has both.

Defaults are k1 = 1.2 and b = 0.75 in both Lucene and Elasticsearch. The monograph reports that values in the range 0.5 < b < 0.8 and 1.2 < k1 < 2 are “reasonably good in many circumstances”, and that the model itself gives no guidance, so you tune them against judged queries if you tune them at all.

What does BM25 look like in code?

Here it is from scratch, with Lucene’s IDF and default parameters, on four short documents. One of them (d2) is keyword-stuffed on purpose, one (d4) is long and mentions “refund” once near the end.

# bm25.py: Okapi BM25 from scratch. Python 3 standard library only.
import math, re

docs = {
    "d1": "Refunds are issued to the original payment method within 7 days.",
    "d2": "Refund refund refund! Our refund policy page lists every refund rule, refund form and refund FAQ.",
    "d3": "Error ERR-4012 means the card issuer declined the refund. Retry after 24 hours.",
    "d4": ("Shipping takes 3 to 5 days. Orders over 999 rupees ship free. Gift wrapping is available. "
           "Returns are accepted within 30 days. A refund is issued after the item reaches our warehouse "
           "and passes inspection, which usually takes a week."),
}

def tokenize(text):
    return re.findall(r"[a-z0-9]+(?:-[a-z0-9]+)*", text.lower())

toks = {d: tokenize(t) for d, t in docs.items()}
N = len(toks)
avgdl = sum(len(t) for t in toks.values()) / N
df = {}
for t in toks.values():
    for term in set(t):
        df[term] = df.get(term, 0) + 1

def idf(term):  # Lucene's form: never negative
    n = df.get(term, 0)
    return math.log(1 + (N - n + 0.5) / (n + 0.5))

def bm25(query, doc, k1=1.2, b=0.75):
    t, dl = toks[doc], len(toks[doc])
    score = 0.0
    for term in tokenize(query):
        tf = t.count(term)
        if tf:
            score += idf(term) * tf * (k1 + 1) / (tf + k1 * (1 - b + b * dl / avgdl))
    return score

def search(query, **kw):
    ranked = sorted(docs, key=lambda d: bm25(query, d, **kw), reverse=True)
    return "  ".join(f"{d}={bm25(query, d, **kw):.3f}" for d in ranked)

print(f"N={N} avgdl={avgdl:.1f}  idf(refund)={idf('refund'):.3f}  idf(err-4012)={idf('err-4012'):.3f}")
print("q='refund'            ", search("refund"))
print("q='err-4012 refund'   ", search("err-4012 refund"))
print("q='refund' b=0        ", search("refund", b=0))
print("q='refund' k1=100     ", search("refund", k1=100))
print("q='refunded'          ", search("refunded"))
print()
print("tf saturation, k1=1.2, doc of average length:")
for tf in (1, 2, 3, 5, 10, 100):
    print(f"  tf={tf:<3} -> {tf * 2.2 / (tf + 1.2):.3f}  (ceiling 2.2)")

Output (Python 3.14.5):

N=4 avgdl=20.0  idf(refund)=0.357  idf(err-4012)=1.204
q='refund'             d2=0.685  d3=0.416  d4=0.253  d1=0.000
q='err-4012 refund'    d3=1.821  d2=0.685  d4=0.253  d1=0.000
q='refund' b=0         d2=0.670  d3=0.357  d4=0.357  d1=0.000
q='refund' k1=100      d2=2.741  d3=0.482  d4=0.205  d1=0.000
q='refunded'           d1=0.000  d2=0.000  d3=0.000  d4=0.000

tf saturation, k1=1.2, doc of average length:
  tf=1   -> 1.000  (ceiling 2.2)
  tf=2   -> 1.375  (ceiling 2.2)
  tf=3   -> 1.571  (ceiling 2.2)
  tf=5   -> 1.774  (ceiling 2.2)
  tf=10  -> 1.964  (ceiling 2.2)
  tf=100 -> 2.174  (ceiling 2.2)

Five things to read off that output:

  1. Rare terms dominate. err-4012 appears in one document, so its IDF (1.204) is more than three times that of refund (0.357, in three of four documents). Add it to the query and d3 jumps to the top with 1.821.
  2. Saturation works. d2 says “refund” seven times and still only scores 1.6 times what d3 gets for one mention. Turn saturation nearly off with k1=100 and the stuffed page wins by 5.7 times.
  3. Length matters. With b=0, d3 and d4 tie at 0.357 because length is ignored. With the default b=0.75, the long d4 drops to 0.253.
  4. BM25 sees tokens, not words. d1 is the most useful answer and scores zero, because “Refunds” is a different token from “refund”. The query “refunded” matches nothing. Production analyzers stem words to fix part of this; meaning-level matching needs embeddings.
  5. One mention at average length scores exactly IDF. tf=1 gives a tf factor of 1.000, because (k1 + 1) / (1 + k1) is 1.

Why is BM25 still used in RAG?

Because it is good at exactly what embeddings are bad at. Dense vectors compress text into meaning, which blurs identifiers: error codes, SKUs, ticket numbers, function names, rare surnames. BM25 rewards exact rare tokens, as err-4012 shows above. Anthropic’s contextual retrieval experiments found that adding BM25 to contextual embeddings cut the top-20 retrieval failure rate further, from a 35% reduction to 49%.

It is also cheap. An inverted index needs no GPU, no model download and no re-embedding when you change embedding models. It is easy to debug: you can see exactly which terms matched and why a document scored what it did. And because it needs no training data, it is the honest zero-shot baseline. In the HyDE paper, BM25 scored 50.6 nDCG@10 on TREC DL19, ahead of the unsupervised dense retriever Contriever at 44.5.

BM25 vs TF-IDF vs embeddings: what’s the difference?

TF-IDF multiplies raw term frequency by IDF. BM25 is the same family with two corrections: tf saturates (k1) and long documents are discounted (b). Robertson and Zaragoza point out that the saturating tf component is what sets it apart from the common tf or 1 + log tf choices, which never reach a maximum. Elasticsearch ships BM25 as its default similarity.

Embeddings are a different kind of retrieval. They map text to vectors and compare by cosine similarity, so “money back” can match “refund” with no shared words. They miss exact tokens that BM25 catches. The two fail on different queries, which is why combining them works. I cover the combination in hybrid RAG and semantic search, and the fusion step in reciprocal rank fusion.

Where does BM25 break?

  • Vocabulary mismatch. No shared token, no score. Synonyms, paraphrases and other languages all score zero unless your analyzer or query expansion bridges them.
  • Tokenization choices. Whether ERR-4012 is one token or two, whether “Refunds” is stemmed, and how you handle Gujarati or Hindi script decide what can match. The demo above keeps hyphenated codes as one token on purpose.
  • Score scale. BM25 scores are unbounded and depend on the collection. A 14 in one index and a 14 in another mean different things, so you can’t add them to cosine similarities directly. Rank-based fusion sidesteps this.
  • Small or skewed collections. IDF needs enough documents to say what is rare. On a few hundred chunks, IDF is noisy.

What I would do

Keep BM25 in the stack even after adding vectors. In a RAG pipeline I’d index the same chunks into a BM25 index (Elasticsearch, OpenSearch, or a few dozen lines like the demo above for small sets; note that Postgres’s built-in ts_rank is not BM25) and an embedding index, retrieve from both and fuse.

I’d leave k1 and b at the defaults until there is a judged query set to tune against. Time spent on the analyzer (keep codes intact, stem where it helps, handle the scripts your users write in) pays off more than parameter sweeps. And I’d add a reranker after fusion, so BM25 only has to get the right chunk into the shortlist, not to the top.

FAQ

What does BM stand for in BM25?

“Best Matching”, per the common expansion. The “Okapi” in the full name is the search system where the Robertson group first used it; their “Okapi at TREC” papers from 1992 onward are where the weighting schemes were tested.

What are good values for k1 and b?

Start with the Lucene and Elasticsearch defaults, k1 = 1.2 and b = 0.75. Robertson and Zaragoza report 0.5 to 0.8 for b and 1.2 to 2 for k1 as reasonable ranges. Lower b for collections where long documents are long because they cover more.

For exact terms, codes and names, often yes. For paraphrases and conceptual questions, no. Most production RAG systems run both and fuse the rankings.

Does BM25 need training?

No. It uses only collection statistics (document frequencies and lengths), so it works on day one with no labeled data.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.