RAG chunking strategies: how to split documents for RAG
RAG chunking strategies decide how documents are split before embedding. Fixed-size, recursive, structure-aware and semantic chunking compared, with Python.

Chunking is how a RAG system splits documents into the passages it indexes and retrieves, and the strategy decides whether a fact survives as one searchable unit or gets cut in half. The common strategies are fixed-size windows (with or without overlap), recursive splitting on paragraph and sentence boundaries, structure-aware splitting on headings, and semantic chunking that groups sentences by embedding similarity. For most text, recursive or heading-aware chunks of a few hundred tokens, with the document title and section kept on each chunk, are a sound default.
Chunking happens once, at index time, and everything downstream inherits it. The retriever can only return chunks that exist, and the LLM only sees what the retriever returns. If you haven’t read it yet, what RAG is explains where this step sits.
Why does chunking matter so much?
Three reasons, all mechanical.
Embeddings average. An embedding model turns a whole chunk into one vector. A chunk about three topics produces a vector that is a bit like each and close to none. Smaller, focused chunks match focused questions better.
Context is per chunk. A chunk cut from the middle of a section may never name its subject. “It carries over up to 10 days” is useless if “it” was defined two paragraphs earlier. Contextual retrieval exists because of this.
Budget is per chunk. You send the top k chunks to the model. Big chunks mean fewer distinct sources fit in the same token budget, and more irrelevant text rides along with each relevant sentence. My post on the context window covers why that costs you.
Chroma’s July 2024 technical report, Evaluating Chunking Strategies for Retrieval by Brandon Smith and Anton Troynikov, measured this directly. Across their corpora, chunking choices moved recall by “up to 9%”, and they warned that “default settings for certain popular chunking strategies can lead to relatively poor performance”.
What are the main chunking strategies?
Fixed-size. Cut every N tokens, optionally overlapping consecutive chunks by M tokens. Simple, predictable sizes, and blind to sentences, paragraphs and headings.
Recursive. Try to split on the coarsest separator first (blank lines, i.e. paragraphs), then fall back to finer ones (newlines, sentence ends, spaces) only for pieces that are still too long, and merge small neighbors back up to the size limit. LangChain’s RecursiveCharacterTextSplitter is the well-known version; its source defaults to the separators ["\n\n", "\n", " ", ""].
Structure-aware. Use the document’s own structure: Markdown or HTML headings, PDF sections, code functions and classes, table rows. Each chunk is a section (split further if it is too long), and the heading path travels with it.
Semantic. Embed each sentence and start a new chunk where the similarity between neighbors drops, or cluster sentences into chunks. Chroma’s report tests the KamradtSemanticChunker and its own ClusterSemanticChunker.
LLM-based. Ask a model to choose split points. Chroma tested a GPT-4o chunker, which had the highest recall in their runs (91.9%) and among the lowest precision (3.9%), and it means an LLM call over every document at index time.
There are also strategies that change what gets embedded rather than where the cuts go: late chunking (Günther et al., September 2024) runs the whole document through a long-context embedding model first and pools token embeddings per chunk afterwards, so each chunk vector carries context from the rest of the document. Contextual retrieval does something similar with an LLM-written prefix.
What does each strategy do to a real document?
Here are four configurations on a short Markdown policy. The test is whether the answer to “how many days carry over?” (“up to 10 days”) ends up whole inside one chunk. Sizes are in words to avoid a tokenizer dependency; real pipelines count tokens.
# chunking.py: three chunkers on one document. Python 3 stdlib only.
# Sizes are in words to keep it dependency-free; real pipelines count tokens.
import re
doc = """# Leave policy
Employees get 24 days of paid leave per year. Leave accrues monthly, at 2 days per month.
## Carry over
Unused leave carries over to the next year, up to 10 days. Anything above 10 days lapses on 31 March. Carried-over days are used first.
## Requests
Send leave requests to your manager in the HR portal at least 5 working days ahead. For more than 10 consecutive days, HR must also approve."""
FACT = "up to 10 days" # the answer to "how many days carry over?"
def fixed(text, size=16, overlap=0):
w, step = text.split(), size - overlap
return [" ".join(w[i:i + size]) for i in range(0, len(w), step) if w[i:i + size]]
def recursive(text, size=30, seps=("\n\n", "\n", ". ", " ")):
# Split on the coarsest separator; recurse into any piece that is still too long,
# then greedily merge neighbours back up to the size limit (LangChain's idea).
if len(text.split()) <= size:
return [text.strip()]
sep = next((s for s in seps if s in text), " ")
parts, out, cur = [p for p in text.split(sep) if p.strip()], [], ""
for p in parts:
cand = f"{cur}{sep}{p}" if cur else p
if len(cand.split()) <= size:
cur = cand
else:
if cur: out.append(cur.strip())
if len(p.split()) > size:
out += recursive(p, size, seps[seps.index(sep) + 1:]); cur = ""
else:
cur = p
if cur: out.append(cur.strip())
return out
def by_heading(text):
# Structure-aware: one chunk per section, with the heading path prepended.
out, title = [], ""
for block in re.split(r"\n(?=#)", text):
head, _, body = block.partition("\n")
if head.startswith("# "): title = head[2:]
label = title if head.startswith("# ") else f"{title} > {head.lstrip('# ')}"
if body.strip(): out.append(f"{label}: {' '.join(body.split())}")
return out
for name, chunks in [("fixed 16, no overlap", fixed(doc)),
("fixed 16, overlap 4", fixed(doc, 16, 4)),
("recursive 30", recursive(doc)),
("by heading", by_heading(doc))]:
whole = sum(FACT in " ".join(c.split()) for c in chunks)
print(f"{name:<22} {len(chunks)} chunks, fact intact in {whole}")
for c in chunks:
print(" |", " ".join(c.split()))
Output (Python 3.14.5):
fixed 16, no overlap 5 chunks, fact intact in 0
| # Leave policy Employees get 24 days of paid leave per year. Leave accrues monthly, at
| 2 days per month. ## Carry over Unused leave carries over to the next year, up
| to 10 days. Anything above 10 days lapses on 31 March. Carried-over days are used first.
| ## Requests Send leave requests to your manager in the HR portal at least 5 working
| days ahead. For more than 10 consecutive days, HR must also approve.
fixed 16, overlap 4 7 chunks, fact intact in 1
| # Leave policy Employees get 24 days of paid leave per year. Leave accrues monthly, at
| Leave accrues monthly, at 2 days per month. ## Carry over Unused leave carries over to
| leave carries over to the next year, up to 10 days. Anything above 10 days lapses
| above 10 days lapses on 31 March. Carried-over days are used first. ## Requests Send leave
| ## Requests Send leave requests to your manager in the HR portal at least 5 working
| at least 5 working days ahead. For more than 10 consecutive days, HR must also approve.
| HR must also approve.
recursive 30 3 chunks, fact intact in 1
| # Leave policy Employees get 24 days of paid leave per year. Leave accrues monthly, at 2 days per month. ## Carry over
| Unused leave carries over to the next year, up to 10 days. Anything above 10 days lapses on 31 March. Carried-over days are used first. ## Requests
| Send leave requests to your manager in the HR portal at least 5 working days ahead. For more than 10 consecutive days, HR must also approve.
by heading 3 chunks, fact intact in 1
| Leave policy: Employees get 24 days of paid leave per year. Leave accrues monthly, at 2 days per month.
| Leave policy > Carry over: Unused leave carries over to the next year, up to 10 days. Anything above 10 days lapses on 31 March. Carried-over days are used first.
| Leave policy > Requests: Send leave requests to your manager in the HR portal at least 5 working days ahead. For more than 10 consecutive days, HR must also approve.
What each one shows:
- Fixed 16, no overlap cut the answer in half: “…to the next year, up” and “to 10 days.” Neither chunk contains the fact, so no retriever can return it whole. This is the classic fixed-size failure.
- Fixed 16, overlap 4 recovers the fact in one chunk, at the cost of 7 chunks instead of 5 (40% more to embed and store) and a stray 4-word chunk at the end. Overlap is a patch for blind cuts.
- Recursive 30 keeps sentences and paragraphs whole, but look where the headings went. “## Carry over” is glued to the end of the first chunk, and the carry-over chunk itself has no heading. A greedy merge packs the next small piece into the current chunk, and a heading is a small piece.
- By heading keeps each section whole and prefixes the path (“Leave policy > Carry over”). The chunk that holds the answer also says what it is about, which helps both BM25 and embeddings.
What chunk size should I use?
There is no universal number, but there is evidence for “smaller than most defaults”. In Chroma’s report, RecursiveCharacterTextSplitter at 200 tokens with no overlap was “consistently high performing across all evaluation metrics” (recall 88.1% with text-embedding-3-large), while the OpenAI Assistants file-search default they tested, 800-token chunks with 400 overlap, showed “particularly poor recall-efficiency tradeoffs” and the lowest precision and IoU in their table. Their semantic ClusterSemanticChunker at 200 tokens had the best precision and IoU.
Anthropic’s contextual retrieval post describes chunks as “usually no more than a few hundred tokens” and uses 800-token chunks in its cost example. The original RAG paper used disjoint 100-word chunks of Wikipedia.
The other finding from Chroma worth remembering: the best setting depended on the embedding model. With the small all-MiniLM-L6-v2, 250-token chunks with 125 overlap did best, and dropping the overlap cut recall from 82.4% to 77.1%. Chunk size and embedding model are one decision, not two.
Where does chunking break?
- Tables. Splitting a table by tokens separates values from their column headers. Chunk by row and repeat the header, or convert rows to sentences.
- Code. Splitting mid-function produces chunks that compile to nothing and embed poorly. Split on functions and classes.
- PDFs. Extracted text loses headings, merges columns and puts footers in the middle of paragraphs. Chunking quality starts at parsing quality.
- Cross-references. “See section 4.2” and pronouns pointing at earlier paragraphs break whatever you do. Contextual prefixes help; nothing fixes it fully.
- Re-chunking cost. Changing strategy means re-embedding the corpus. On a large index that is a real migration, so evaluate before you commit. From my years on data platforms at Walmart, I’d treat chunker version as part of the index schema and store it on every chunk.
What I would do
- Parse well first: keep headings, list structure and tables from the source format.
- Chunk on structure (headings, then paragraphs, then sentences), targeting roughly 200 to 400 tokens, and prefix every chunk with the document title and heading path.
- Use overlap only when you are forced into fixed-size cuts.
- Store
doc_id, section path, position and chunker version with each chunk, so you can fetch neighbors at answer time and re-chunk later without guessing. - Pick the chunk size together with the embedding model, on a judged query set, using recall at k. How to evaluate RAG and how to choose an embedding model cover the measuring.
- Then add hybrid retrieval and contextual prefixes, which fix what chunking alone cannot.
FAQ
What is the best chunk size for RAG?
It depends on your documents and embedding model, but a few hundred tokens is the usual range. Chroma’s 2024 evaluation found 200-token recursive chunks with no overlap consistently strong, and an 800-token, 400-overlap default weak. Measure on your own queries.
Should RAG chunks overlap?
Overlap helps when cuts ignore sentence and section boundaries, because it keeps facts that straddle a cut in at least one chunk. With structure-aware splitting, it matters less. It always costs more storage and embedding calls.
What is semantic chunking?
Splitting text where the meaning shifts, detected by comparing embeddings of neighboring sentences, instead of at a fixed length. It can improve precision but costs an embedding call per sentence at index time.
What is the difference between chunking and tokenization?
Tokenization splits text into the units a model reads (tokens). Chunking splits documents into retrievable passages, each of which is many tokens.
Sources
- Brandon Smith and Anton Troynikov, Chroma, Evaluating Chunking Strategies for Retrieval, July 3, 2024
- LangChain,
RecursiveCharacterTextSplittersource (read 2026-10-31) - Anthropic, Introducing Contextual Retrieval, September 19, 2024
- Günther, Mohr, Williams, Wang and Xiao, Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models, arXiv September 7, 2024
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS 2020