What is ColBERT? Late interaction retrieval, explained
ColBERT is a late interaction retriever: one vector per token and a MaxSim score instead of one vector per document. How it works and costs, with Python code.

ColBERT is a retrieval model that keeps one embedding per token instead of squeezing a whole passage into one vector, and scores a document by matching each query token to its most similar document token and adding up those best matches (an operation called MaxSim). Because documents are encoded ahead of time and queries separately, the expensive work happens offline, which is what “late interaction” means. Omar Khattab and Matei Zaharia at Stanford introduced it at SIGIR 2020, and ColBERTv2 (NAACL 2022) made its storage practical.
What problem does ColBERT solve?
Neural retrieval had two camps, and each paid a price.
Bi-encoders (single-vector dense retrieval, the default in most vector databases) embed the query and each document into one vector each and compare them with a dot product. Document vectors are computed once, so search is fast. But one vector has to summarize the whole passage, and details get averaged away.
Cross-encoders feed the query and a document through BERT together, so every query word can attend to every document word. They are the most accurate and they power rerankers, but nothing can be precomputed: each query needs one full model pass per candidate. The ColBERT paper notes that BERT-based rankers of the time were 100 to 1,000 times more computationally expensive than earlier models, and that BERT raised MRR@10 on MS MARCO by almost 7% while adding up to tens of thousands of milliseconds of latency.
ColBERT sits between them. It keeps the per-token detail of interaction models and the precomputation of bi-encoders. The paper reports that in re-ranking, ColBERT was over 170 times faster and needed 14,000 times fewer FLOPs than existing BERT-based models while beating every non-BERT baseline, and the abstract summarizes it as two orders of magnitude faster with four orders of magnitude fewer FLOPs per query.
How does ColBERT’s late interaction work?
Three steps.
- Encode documents offline. Each passage goes through a BERT encoder. Every output token becomes a vector, passed through a linear layer to a smaller size (the paper uses 128 dimensions) and normalized to length 1. Punctuation embeddings are filtered out. All of these vectors are stored.
- Encode the query at search time. The query is padded with
[MASK]tokens to a fixed length (32 in the paper). The authors call this query augmentation: BERT produces useful embeddings at the masked positions that act as soft, learned query expansion, and their ablations show it matters. - Score with MaxSim. For each query token vector, find the maximum cosine similarity with any document token vector. Sum those maxima over the query tokens. In the paper’s notation,
S(q, d) = sum over i of (max over j of dot(Eq_i, Ed_j)).
The interaction between query and document happens only in step 3, after both sides were encoded independently. That is the “late” part, and it is why document vectors can be computed once and reused for every query.
What does MaxSim look like in code?
Here is a single-vector score and a MaxSim score on the same two documents. The token vectors come from a hand-written table so the example runs offline; in ColBERT they are contextual BERT outputs, so the same word gets a different vector in a different sentence.
# maxsim.py: one vector per document vs ColBERT-style late interaction (MaxSim).
# Python 3 stdlib only. Token vectors come from a hand-written table so this runs offline;
# in ColBERT they are contextual BERT outputs, 128 dims, normalized to length 1.
import math
DIMS = ["python", "java", "memory", "bug", "web", "testing", "deploy"]
TOK = { # token -> loading on each dim, before normalization
"python": (1, .2, 0, 0, 0, 0, 0), "java": (.2, 1, 0, 0, 0, 0, 0),
"memory": (0, 0, 1, .2, 0, 0, 0), "leak": (0, 0, .7, .7, 0, 0, 0),
"heap": (0, 0, .9, 0, 0, 0, 0), "django": (.6, 0, 0, 0, .8, 0, 0),
"routing": (0, 0, 0, 0, 1, 0, 0), "pytest": (.5, 0, 0, 0, 0, .9, 0),
"fixtures": (0, 0, 0, 0, 0, 1, 0), "docker": (0, 0, 0, 0, 0, 0, 1),
"kubernetes": (0, 0, 0, 0, 0, 0, 1), "templates": (0, 0, 0, 0, .9, 0, 0),
}
def unit(v): n = math.sqrt(sum(x * x for x in v)); return [x / n for x in v]
def dot(a, b): return sum(x * y for x, y in zip(a, b))
def enc(text): return [unit(TOK[t]) for t in text.split()]
docs = {
"A python handbook": "python django routing templates pytest fixtures docker kubernetes memory leak",
"B java heap guide": "java memory leak heap",
}
query = "python memory leak"
Q = enc(query)
print(f"query: {query!r}")
for name, text in docs.items():
D = enc(text)
pooled = unit([sum(col) for col in zip(*D)]) # mean-pool to ONE vector
single = dot(unit([sum(col) for col in zip(*Q)]), pooled)
per_tok = [max(dot(q, d) for d in D) for q in Q] # best doc token per query token
print(f" {name:<18} single-vector cos {single:.2f} MaxSim {sum(per_tok):.2f} "
f"(per query token {', '.join(f'{x:.2f}' for x in per_tok)})")
# Storage: what multi-vector costs. Passage count from the ColBERT paper (MS MARCO);
# tokens per passage is my assumption.
passages, toks = 8_800_000, 70
print()
print(f"one 768-d float32 vector per passage: {passages * 768 * 4 / 1e9:6.1f} GB")
print(f"{toks} x 128-d float16 vectors per passage: {passages * toks * 128 * 2 / 1e9:6.1f} GB")
print(f"{toks} x (2-bit residual + 4-byte centroid id): {passages * toks * (128 * 2 / 8 + 4) / 1e9:6.1f} GB")
Output (Python 3.9.6):
query: 'python memory leak'
A python handbook single-vector cos 0.55 MaxSim 3.00 (per query token 1.00, 1.00, 1.00)
B java heap guide single-vector cos 0.88 MaxSim 2.38 (per query token 0.38, 1.00, 1.00)
one 768-d float32 vector per passage: 27.0 GB
70 x 128-d float16 vectors per passage: 157.7 GB
70 x (2-bit residual + 4-byte centroid id): 22.2 GB
The query asks about a Python memory leak. Document A is a broad Python handbook with one line about memory leaks; document B is about Java heap leaks. Mean-pooling A’s ten tokens into one vector dilutes the leak signal under Django, pytest and Docker, so the single-vector score prefers the wrong document, B, by 0.88 to 0.55. MaxSim lets each query token find its own best match. “python”, “memory” and “leak” each find an exact partner in A, for a perfect 3.00, while B can’t match “python” well and scores 2.38. Long, multi-topic passages are exactly where single vectors lose detail, and where late interaction earns its cost.
Why is ColBERT expensive to store, and how does ColBERTv2 fix it?
The bottom half of that output is the catch. A single-vector index stores one vector per passage; ColBERT stores one per token. The ColBERTv2 paper says late interaction multiplies the storage footprint by about an order of magnitude. My 70-tokens-per-passage figure is an assumption for illustration, but with it, MS MARCO’s 8.8 million passages need about 158 GB at 16-bit precision against 27 GB for one 768-dimension float32 vector each.
ColBERTv2 (Santhanam, Khattab, Saad-Falcon, Potts and Zaharia) observed that ColBERT’s token vectors cluster tightly by word sense. So it stores each vector as the ID of its nearest centroid plus a quantized residual. With 128 dimensions, the paper uses 4 bytes for the centroid ID and 16 or 32 bytes for the residual at 1 or 2 bits per dimension, for 20 or 36 bytes per vector instead of 256 bytes at 16-bit precision. The abstract reports a 6 to 10 times smaller footprint with state-of-the-art quality in and out of domain. The last line of my output applies that 36-byte figure, and the multi-vector index drops below the single-vector one.
ColBERTv2 also changed training: it distills scores from a cross-encoder and mines hard negatives, which is a large part of its quality gain. Search runs through PLAID, a retrieval engine described in a CIKM 2022 paper, which the stable branch of the official repository includes.
How does ColBERT search a whole collection?
Scoring with MaxSim is cheap per document, but you still can’t score millions of documents per query. The original paper used a two-stage approach. First, each of the 32 query vectors runs a nearest-neighbor search over all document token vectors in a FAISS IVFPQ index, and the hits are mapped back to their documents. Second, only those candidate documents are scored exactly with full MaxSim. ColBERTv2 does the same with its centroids: find the centroids closest to each query vector, gather the passages whose tokens live there, compute approximate scores, then rescore the best candidates in full.
The BEIR benchmark found late-interaction and re-ranking models had the best average zero-shot performance across 18 datasets, at a higher compute cost than BM25 or single-vector dense retrieval. That matches the intuition from the demo: token-level matching transfers to new domains better than a single learned summary.
ColBERT vs bi-encoder vs cross-encoder: which should I use?
- Bi-encoder (one vector per document): cheapest storage and fastest search. Use it for first-stage retrieval at scale, ideally inside hybrid search.
- ColBERT (one vector per token): better on long or multi-topic passages and out of domain, at a higher storage and engineering cost. Use it as first-stage retrieval when quality matters and you can run the index, or as a reranker over a few hundred candidates.
- Cross-encoder (query and document together): the most accurate and the most expensive per candidate. Use it only to rerank a short list.
What I would do
- Start with BM25 plus a single-vector model and a cross-encoder reranker. It is the simplest system that is hard to beat, and it gives you the labelled baseline you need to judge anything else.
- Try ColBERT when your evaluation shows the first stage missing documents that the reranker would have ranked well, especially long documents covering many topics.
- Use it as a reranker first. Scoring 100 to 500 candidates with MaxSim needs no special index and tells you quickly whether late interaction helps on your data.
- If it helps, move to a full ColBERTv2 or PLAID index, and budget storage per token, not per document.
FAQ
What does ColBERT stand for?
Contextualized Late interaction over BERT, from the original paper’s title.
What is late interaction?
Encoding the query and the document separately, then letting them interact only through a cheap scoring step at the end, such as MaxSim over token vectors. Early interaction, as in cross-encoders, mixes them inside the model.
What is MaxSim?
For each query token vector, the maximum similarity to any document token vector. ColBERT’s relevance score is the sum of these maxima over all query tokens.
Is ColBERT better than dense retrieval?
On many benchmarks, especially out of domain, it scores higher than single-vector retrievers. It also costs more storage and a more complex index, so whether it is better for you depends on your corpus and your quality bar. Measure it on your own queries.
Can I use ColBERT with a normal vector database?
Only partly. MaxSim needs many vectors per document and a per-document aggregation step, which most single-vector stores do not do natively. The common approach is the official ColBERT library with its PLAID index, or using ColBERT as a reranker over candidates from another store.
Sources
- Khattab and Zaharia, ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT, SIGIR 2020
- Santhanam et al., ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction, NAACL 2022
- stanford-futuredata/ColBERT, official repository, checked 2026-10-31
- Thakur et al., BEIR, NeurIPS 2021 Datasets and Benchmarks
- Johnson, Douze and Jégou, Billion-scale similarity search with GPUs, arXiv, February 2017