What is hybrid RAG? BM25 plus vectors, explained
Hybrid RAG combines keyword (BM25) and vector search, then fuses the rankings. Why each misses what the other finds, and how to fuse them, with runnable Python.

Hybrid RAG is retrieval-augmented generation where the retrieval step runs two searches over the same chunks, a keyword search (usually BM25) and a vector search over embeddings, and merges the two ranked lists before the top results go into the prompt. Keyword search catches exact terms such as error codes, names and IDs; vector search catches paraphrases and meaning. Running both covers each one’s blind spot.
The merge step is usually reciprocal rank fusion, which combines lists by rank instead of by score. If you are new to the overall pattern, start with what RAG is; this post is about the retrieval half.
Why isn’t vector search enough?
Embeddings turn text into a point in a vector space, and nearby points mean similar meaning. That is excellent for “how do I get my money back?” matching a chunk about refunds with no shared words. It is weak for tokens whose meaning is their exact spelling. ERR-4012 and ERR-4013 look nearly identical to an embedding model, and so do two SKUs, two invoice numbers or two people with similar names. Support tickets, logs, code, contracts and product catalogs are full of these.
Keyword search has the opposite profile. BM25 scores exact token matches, weighted by how rare the token is, so a query containing ERR-4012 goes straight to the one document that contains it. But “refunded” doesn’t match “refunds” without stemming, and “money back” never matches “refund” at all.
The evidence that the combination helps is consistent. Anthropic’s contextual retrieval write-up measured top-20 retrieval failure across codebases, fiction, arXiv papers and science papers. Contextual embeddings alone cut the failure rate by 35%; adding contextual BM25 and fusing the lists took that to 49%; reranking on top reached 67%. Their description of the pipeline is a good definition of hybrid retrieval: run BM25 and embeddings, then “combine and deduplicate results using rank fusion techniques”.
How does hybrid RAG work?
The steps:
- Index twice. Each chunk goes into an inverted index for BM25 and into a vector database as an embedding. Same chunk IDs in both.
- Retrieve from both. Run the user’s query against each index and keep the top N from each, typically a few dozen.
- Fuse. Merge the two lists into one ranking. RRF gives each chunk
1 / (60 + rank)per list and sums them, so chunks that both retrievers like rise to the top. - Rerank (optional, recommended). Pass the fused top candidates through a cross-encoder reranker that reads the query and each chunk together.
- Generate. Put the final top few chunks in the prompt.
What does hybrid retrieval look like in code?
This script runs BM25 and a vector-style retriever over five support articles and fuses them with RRF. I can’t ship a real embedding model in standard-library Python, so the vector side is a character-trigram cosine. It is a stand-in: it matches word variants (“refunded” and “refunds” share trigrams) the way an embedding matches paraphrases, and it blurs near-identical strings the way embeddings do.
# hybrid.py: BM25 + a vector-style retriever, fused with RRF. Python 3 stdlib only.
# The "vector" side is a character-trigram cosine, a stand-in for an embedding model:
# it matches word variants ("refunded" ~ "refunds") the way dense retrieval matches meaning.
import math, re
from collections import Counter
docs = {
"refunds": "Refunds are issued to the original payment method within 7 days.",
"err-4012": "Error ERR-4012 means the card issuer declined the transaction. Retry after 24 hours.",
"err-4013": "Error ERR-4013 means the payment gateway timed out before confirming.",
"returns": "Returned items are inspected at the warehouse before any money goes back.",
"shipping": "Orders over 999 rupees ship free. Shipping takes 3 to 5 days.",
}
def words(t): return re.findall(r"[a-z0-9]+(?:-[a-z0-9]+)*", t.lower())
W = {d: words(t) for d, t in docs.items()}
N, avgdl = len(W), sum(map(len, W.values())) / len(W)
df = Counter(w for ws in W.values() for w in set(ws))
def bm25(q, d, k1=1.2, b=0.75):
s = 0.0
for w in words(q):
tf = W[d].count(w)
if tf:
idf = math.log(1 + (N - df[w] + 0.5) / (df[w] + 0.5))
s += idf * tf * (k1 + 1) / (tf + k1 * (1 - b + b * len(W[d]) / avgdl))
return s
def trigrams(t):
t = f" {t.lower()} "
return Counter(t[i:i + 3] for i in range(len(t) - 2))
V = {d: trigrams(t) for d, t in docs.items()}
def cosine(q, d):
a, b = trigrams(q), V[d]
dot = sum(a[g] * b[g] for g in a)
return dot / (math.sqrt(sum(x * x for x in a.values())) * math.sqrt(sum(x * x for x in b.values())))
def ranked(fn, q, k=3):
hits = [(d, fn(q, d)) for d in docs]
return [h for h in sorted(hits, key=lambda x: x[1], reverse=True) if h[1] > 0][:k]
def rrf(*lists, k=60):
s = Counter()
for lst in lists:
for r, (d, _) in enumerate(lst, 1):
s[d] += 1 / (k + r)
return s.most_common(3)
for q in ["ERR-4012", "when do I get refunded", "money back on returned items"]:
kw, vec = ranked(bm25, q), ranked(cosine, q)
print(f"query: {q!r}")
print(" bm25 :", ", ".join(f"{d} {s:.2f}" for d, s in kw) or "(no hits)")
print(" vector:", ", ".join(f"{d} {s:.2f}" for d, s in vec))
print(" hybrid:", ", ".join(f"{d} {s:.4f}" for d, s in rrf(kw, vec)))
Output (Python 3.14.5):
query: 'ERR-4012'
bm25 : err-4012 1.32
vector: err-4012 0.35, err-4013 0.33
hybrid: err-4012 0.0328, err-4013 0.0161
query: 'when do I get refunded'
bm25 : (no hits)
vector: refunds 0.15, returns 0.07, err-4012 0.04
hybrid: refunds 0.0164, returns 0.0161, err-4012 0.0159
query: 'money back on returned items'
bm25 : returns 5.47
vector: returns 0.47, err-4012 0.09, refunds 0.04
hybrid: returns 0.0328, err-4012 0.0161, refunds 0.0159
Three queries, three behaviors:
ERR-4012: the vector side scoreserr-4012anderr-4013at 0.35 and 0.33, nearly a tie, which is exactly the failure that makes people add keyword search. BM25 only matches the right one. Fused, the correct article leads by a factor of two because both retrievers agree on it.- “when do I get refunded”: BM25 returns nothing, because no document contains “refunded”. The vector side finds
refunds. Hybrid still answers, because one working retriever is enough to get a document into the fused list. - “money back on returned items”: both agree on
returns, so it wins clearly. The fused tail (err-4012,refunds) comes from the vector side’s weak matches. That tail is why I put a reranker after fusion.
How should the scores be combined?
There are two families. Rank fusion (RRF) ignores scores and sums 1 / (k + rank). Score fusion normalizes each list’s scores (for example min-max to 0 to 1) and takes a weighted sum, often with a parameter like Weaviate’s alpha, where 0 is keyword-only and 1 is vector-only. Weaviate’s hybrid search docs list a default alpha of 0.75 and relativeScoreFusion as the default fusion since v1.24.
What you must not do is add raw scores. BM25 scores are unbounded and collection-dependent, cosine similarities sit in a narrow band, and the bigger numbers silently win. I show that bug with numbers in the RRF post. Elasticsearch’s rrf retriever does rank fusion server-side with rank_constant defaulting to 60.
Where does hybrid RAG break?
- Two indexes to keep in sync. Every insert, update and delete has to hit both. A chunk that exists in one and not the other gets half its votes. From years on data platforms at Walmart, this is the part I’d budget for: the dual-write and the backfill, not the query code.
- Tokenization still matters. If your BM25 analyzer splits
ERR-4012intoerrand4012, keyword search loses most of its advantage on identifiers. - Fusion can’t fix bad candidates. If neither retriever surfaces the right chunk in its top N, fusion can’t invent it. Measure recall at N per retriever.
- More latency. Two searches plus fusion is more work than one, though the searches run in parallel and fusion is trivial.
- Chunks without context. Both retrievers see the same chunk text. A chunk that never names its subject is hard for both; that is what contextual retrieval addresses.
What I would do
Hybrid is the first retrieval upgrade I’d make to any RAG system that touches identifiers, which in enterprise data is nearly all of them.
- Store chunks once, with stable IDs, and index them into BM25 and vectors from the same pipeline.
- Retrieve 20 to 50 from each, fuse with RRF at k = 60.
- Rerank the fused top 50 or so with a cross-encoder and keep 5 to 10.
- Evaluate each retriever alone and the fused list on the same judged queries, so you know which side is carrying which queries. How to evaluate RAG has the metrics.
If your stack is Postgres, you can do both sides in one database with full-text search and pgvector, and fuse in SQL or application code.
FAQ
What is the difference between hybrid search and hybrid RAG?
Hybrid search is the retrieval technique: keyword plus vector, fused. Hybrid RAG is a RAG system that uses hybrid search as its retrieval step.
Is hybrid search better than vector search?
On most mixed corpora, yes, because the two fail on different queries. Anthropic’s tests found adding contextual BM25 to contextual embeddings cut retrieval failures further, from a 35% reduction to 49%. Measure on your own queries.
What is the best way to combine BM25 and vector scores?
Reciprocal rank fusion if you have no labeled data to tune with. A weighted sum of normalized scores if you do. Never add raw scores.
Do I need a separate search engine for BM25?
Not necessarily. Elasticsearch, OpenSearch and Weaviate support both sides in one system, and Postgres can pair full-text search with pgvector.
Sources
- Anthropic, Introducing Contextual Retrieval, September 19, 2024
- Cormack, Clarke and Büttcher, Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods, SIGIR 2009
- Robertson and Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond, 2009
- Weaviate, Hybrid search concepts (read 2026-10-31)
- Elastic, Reciprocal rank fusion (read 2026-10-31)