Malhar Ujawane

Blog

Explainer·(updated October 31, 2026)

What is hybrid RAG? BM25 plus vectors, explained

Hybrid RAG combines keyword (BM25) and vector search, then fuses the rankings. Why each misses what the other finds, and how to fuse them, with runnable Python.

What is hybrid RAG? BM25 plus vectors, explained

Hybrid RAG is retrieval-augmented generation where the retrieval step runs two searches over the same chunks, a keyword search (usually BM25) and a vector search over embeddings, and merges the two ranked lists before the top results go into the prompt. Keyword search catches exact terms such as error codes, names and IDs; vector search catches paraphrases and meaning. Running both covers each one’s blind spot.

The merge step is usually reciprocal rank fusion, which combines lists by rank instead of by score. If you are new to the overall pattern, start with what RAG is; this post is about the retrieval half.

Why isn’t vector search enough?

Embeddings turn text into a point in a vector space, and nearby points mean similar meaning. That is excellent for “how do I get my money back?” matching a chunk about refunds with no shared words. It is weak for tokens whose meaning is their exact spelling. ERR-4012 and ERR-4013 look nearly identical to an embedding model, and so do two SKUs, two invoice numbers or two people with similar names. Support tickets, logs, code, contracts and product catalogs are full of these.

Keyword search has the opposite profile. BM25 scores exact token matches, weighted by how rare the token is, so a query containing ERR-4012 goes straight to the one document that contains it. But “refunded” doesn’t match “refunds” without stemming, and “money back” never matches “refund” at all.

The evidence that the combination helps is consistent. Anthropic’s contextual retrieval write-up measured top-20 retrieval failure across codebases, fiction, arXiv papers and science papers. Contextual embeddings alone cut the failure rate by 35%; adding contextual BM25 and fusing the lists took that to 49%; reranking on top reached 67%. Their description of the pipeline is a good definition of hybrid retrieval: run BM25 and embeddings, then “combine and deduplicate results using rank fusion techniques”.

How does hybrid RAG work?

Hybrid retrieval pipelineA query on the left fans out to two retrievers: BM25 keyword search over an inverted index, and vector search over an embedding index. Each returns its own ranked list. Both lists feed a fusion step (reciprocal rank fusion), shown in orange, which feeds an optional reranker, which passes the top few chunks into the LLM prompt on the right.Hybrid retrieval: two searches, one listqueryBM25exact terms, codesvector searchmeaning, paraphrasetop 50top 50fuse (RRF)1/(60+rank)rerankLLMSame chunks, two indexes. Fusion uses ranks, so BM25 and cosine scores never need a common scale.

The steps:

  1. Index twice. Each chunk goes into an inverted index for BM25 and into a vector database as an embedding. Same chunk IDs in both.
  2. Retrieve from both. Run the user’s query against each index and keep the top N from each, typically a few dozen.
  3. Fuse. Merge the two lists into one ranking. RRF gives each chunk 1 / (60 + rank) per list and sums them, so chunks that both retrievers like rise to the top.
  4. Rerank (optional, recommended). Pass the fused top candidates through a cross-encoder reranker that reads the query and each chunk together.
  5. Generate. Put the final top few chunks in the prompt.

What does hybrid retrieval look like in code?

This script runs BM25 and a vector-style retriever over five support articles and fuses them with RRF. I can’t ship a real embedding model in standard-library Python, so the vector side is a character-trigram cosine. It is a stand-in: it matches word variants (“refunded” and “refunds” share trigrams) the way an embedding matches paraphrases, and it blurs near-identical strings the way embeddings do.

# hybrid.py: BM25 + a vector-style retriever, fused with RRF. Python 3 stdlib only.
# The "vector" side is a character-trigram cosine, a stand-in for an embedding model:
# it matches word variants ("refunded" ~ "refunds") the way dense retrieval matches meaning.
import math, re
from collections import Counter

docs = {
    "refunds":   "Refunds are issued to the original payment method within 7 days.",
    "err-4012":  "Error ERR-4012 means the card issuer declined the transaction. Retry after 24 hours.",
    "err-4013":  "Error ERR-4013 means the payment gateway timed out before confirming.",
    "returns":   "Returned items are inspected at the warehouse before any money goes back.",
    "shipping":  "Orders over 999 rupees ship free. Shipping takes 3 to 5 days.",
}

def words(t): return re.findall(r"[a-z0-9]+(?:-[a-z0-9]+)*", t.lower())
W = {d: words(t) for d, t in docs.items()}
N, avgdl = len(W), sum(map(len, W.values())) / len(W)
df = Counter(w for ws in W.values() for w in set(ws))

def bm25(q, d, k1=1.2, b=0.75):
    s = 0.0
    for w in words(q):
        tf = W[d].count(w)
        if tf:
            idf = math.log(1 + (N - df[w] + 0.5) / (df[w] + 0.5))
            s += idf * tf * (k1 + 1) / (tf + k1 * (1 - b + b * len(W[d]) / avgdl))
    return s

def trigrams(t):
    t = f"  {t.lower()}  "
    return Counter(t[i:i + 3] for i in range(len(t) - 2))
V = {d: trigrams(t) for d, t in docs.items()}

def cosine(q, d):
    a, b = trigrams(q), V[d]
    dot = sum(a[g] * b[g] for g in a)
    return dot / (math.sqrt(sum(x * x for x in a.values())) * math.sqrt(sum(x * x for x in b.values())))

def ranked(fn, q, k=3):
    hits = [(d, fn(q, d)) for d in docs]
    return [h for h in sorted(hits, key=lambda x: x[1], reverse=True) if h[1] > 0][:k]

def rrf(*lists, k=60):
    s = Counter()
    for lst in lists:
        for r, (d, _) in enumerate(lst, 1):
            s[d] += 1 / (k + r)
    return s.most_common(3)

for q in ["ERR-4012", "when do I get refunded", "money back on returned items"]:
    kw, vec = ranked(bm25, q), ranked(cosine, q)
    print(f"query: {q!r}")
    print("  bm25  :", ", ".join(f"{d} {s:.2f}" for d, s in kw) or "(no hits)")
    print("  vector:", ", ".join(f"{d} {s:.2f}" for d, s in vec))
    print("  hybrid:", ", ".join(f"{d} {s:.4f}" for d, s in rrf(kw, vec)))

Output (Python 3.14.5):

query: 'ERR-4012'
  bm25  : err-4012 1.32
  vector: err-4012 0.35, err-4013 0.33
  hybrid: err-4012 0.0328, err-4013 0.0161
query: 'when do I get refunded'
  bm25  : (no hits)
  vector: refunds 0.15, returns 0.07, err-4012 0.04
  hybrid: refunds 0.0164, returns 0.0161, err-4012 0.0159
query: 'money back on returned items'
  bm25  : returns 5.47
  vector: returns 0.47, err-4012 0.09, refunds 0.04
  hybrid: returns 0.0328, err-4012 0.0161, refunds 0.0159

Three queries, three behaviors:

  • ERR-4012: the vector side scores err-4012 and err-4013 at 0.35 and 0.33, nearly a tie, which is exactly the failure that makes people add keyword search. BM25 only matches the right one. Fused, the correct article leads by a factor of two because both retrievers agree on it.
  • “when do I get refunded”: BM25 returns nothing, because no document contains “refunded”. The vector side finds refunds. Hybrid still answers, because one working retriever is enough to get a document into the fused list.
  • “money back on returned items”: both agree on returns, so it wins clearly. The fused tail (err-4012, refunds) comes from the vector side’s weak matches. That tail is why I put a reranker after fusion.

How should the scores be combined?

There are two families. Rank fusion (RRF) ignores scores and sums 1 / (k + rank). Score fusion normalizes each list’s scores (for example min-max to 0 to 1) and takes a weighted sum, often with a parameter like Weaviate’s alpha, where 0 is keyword-only and 1 is vector-only. Weaviate’s hybrid search docs list a default alpha of 0.75 and relativeScoreFusion as the default fusion since v1.24.

What you must not do is add raw scores. BM25 scores are unbounded and collection-dependent, cosine similarities sit in a narrow band, and the bigger numbers silently win. I show that bug with numbers in the RRF post. Elasticsearch’s rrf retriever does rank fusion server-side with rank_constant defaulting to 60.

Where does hybrid RAG break?

  • Two indexes to keep in sync. Every insert, update and delete has to hit both. A chunk that exists in one and not the other gets half its votes. From years on data platforms at Walmart, this is the part I’d budget for: the dual-write and the backfill, not the query code.
  • Tokenization still matters. If your BM25 analyzer splits ERR-4012 into err and 4012, keyword search loses most of its advantage on identifiers.
  • Fusion can’t fix bad candidates. If neither retriever surfaces the right chunk in its top N, fusion can’t invent it. Measure recall at N per retriever.
  • More latency. Two searches plus fusion is more work than one, though the searches run in parallel and fusion is trivial.
  • Chunks without context. Both retrievers see the same chunk text. A chunk that never names its subject is hard for both; that is what contextual retrieval addresses.

What I would do

Hybrid is the first retrieval upgrade I’d make to any RAG system that touches identifiers, which in enterprise data is nearly all of them.

  1. Store chunks once, with stable IDs, and index them into BM25 and vectors from the same pipeline.
  2. Retrieve 20 to 50 from each, fuse with RRF at k = 60.
  3. Rerank the fused top 50 or so with a cross-encoder and keep 5 to 10.
  4. Evaluate each retriever alone and the fused list on the same judged queries, so you know which side is carrying which queries. How to evaluate RAG has the metrics.

If your stack is Postgres, you can do both sides in one database with full-text search and pgvector, and fuse in SQL or application code.

FAQ

What is the difference between hybrid search and hybrid RAG?

Hybrid search is the retrieval technique: keyword plus vector, fused. Hybrid RAG is a RAG system that uses hybrid search as its retrieval step.

On most mixed corpora, yes, because the two fail on different queries. Anthropic’s tests found adding contextual BM25 to contextual embeddings cut retrieval failures further, from a 35% reduction to 49%. Measure on your own queries.

What is the best way to combine BM25 and vector scores?

Reciprocal rank fusion if you have no labeled data to tune with. A weighted sum of normalized scores if you do. Never add raw scores.

Do I need a separate search engine for BM25?

Not necessarily. Elasticsearch, OpenSearch and Weaviate support both sides in one system, and Postgres can pair full-text search with pgvector.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.