Chunking strategy is the way source documents are split into retrievable units for embedding, indexing, and RAG. Chunk size, overlap, boundaries, metadata, and hierarchy all affect retrieval quality.
Bad chunking creates bad context. Chunks that are too small may lose meaning. Chunks that are too large may dilute relevance. Evaluate chunking by measuring retrieval quality and downstream answer quality, not by guessing a universal token size.
The decision happens once, at ingestion, and constrains everything after it. No reranker repairs a chunk cut through the middle of the sentence containing the answer.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- A chunk does two jobs at once: it is the unit that gets matched against a query and the unit the model reads. Small chunks match precisely, large chunks explain fully, and one size has to serve both.
- Split on document structure before reaching for a token count. Headings, sections, and list boundaries are semantic boundaries someone already wrote for you.
- Attach metadata at ingestion. Source, title, section heading, date, and permissions are what make filtering, citation, and freshness checks possible later.
- Overlap is cheap insurance against boundary loss, not a fix for boundaries in the wrong place.
- No chunk size is correct in the abstract. Pick two or three candidates and measure them on your own corpus and your own questions.
Why the unit is hard to get right
Retrieval favors small chunks. A tight passage about one idea produces an embedding close to a query about that idea, with little averaged-in noise. Split too far and you get fragments: a chunk reading “This does not apply to trial accounts” is unusable without whatever “this” refers to, and it will still match queries about trial accounts and mislead the model.
Generation favors large chunks. The model needs the heading the passage sits under, the condition that precedes it, the exception that follows. Grow too far and one chunk covers four topics, its embedding averages toward nothing in particular, and it stops matching the specific query that needed it.
The token budget forces the trade. Five chunks at 1,000 tokens each cost 5,000 tokens whether or not the answer needed 200 of them.
The main strategies
Fixed-size with overlap. Split every N tokens with an overlap of M. Trivial to implement, predictable in cost, blind to meaning, and it will cut through the middle of a table. As a baseline to measure against, it is genuinely useful.
Structural, or recursive, splitting. Split on the document’s own hierarchy first: headings, then paragraphs, then sentences, falling back to a token limit only when a section runs long. This is the right default for anything with real structure, which is most documentation, policy, and knowledge base content. It preserves boundaries the author already decided on.
Semantic chunking. Embed sentence by sentence and cut where consecutive sentences diverge past a threshold. It handles unstructured prose such as transcripts and narrative reports where there are no headings to lean on, at the cost of an embedding pass over every sentence and threshold tuning that is fiddly.
Layout-aware splitting. For PDFs, slides, and scanned documents, chunk from parsed layout rather than from extracted text. Tables stay whole with their headers, columns do not interleave, captions stay with their figures. If your corpus is PDF-heavy, this usually moves retrieval more than any embedding change.
Agentic chunking. A model reads the document and decides the boundaries, often writing a summary or likely questions for each chunk as well. Boundary quality is typically the best of the group on messy documents. The cost is an LLM pass over the corpus at ingestion, repeated on every reprocess, plus nondeterminism that makes two runs produce different indexes. Reasonable for a small, high-value, slow-changing corpus, expensive for millions of documents.
Metadata and hierarchy
The chunk you match against does not have to be the chunk you send. Small-to-big retrieval indexes tight passages for matching, then expands each hit to its parent section before assembly. Precise matching, complete context, paid for in storage and tokens. Prepending the document title and section heading to each chunk before embedding is a cheaper version of the same idea, and it gives an orphaned passage the context its position in the document used to supply.
Store metadata alongside every chunk regardless of strategy. Source URI, section, last modified date, and access scope are what let you filter by permission at query time, cite accurately, and detect a stale chunk. Retrofitting metadata means reindexing.
Choosing by measurement
Treat a chunking change like any other pipeline change: fix a question set, run the candidate configurations, and compare per-stage scores rather than a single accuracy number. Benchmarking of chunk size against RAG performance shows how that comparison is structured, though the answer for your corpus will be your own.
Two things make it honest. Use questions drawn from real traffic, since questions generated from a chunk are answerable by that chunk and will flatter whatever chunking produced them. And read the retrieved text on the failures, not only the scores, because the signature of a chunking problem is a passage that is topically correct and cut in the wrong place. Seeing that requires tracing that captures the retrieved chunks on each request. For guidance on testing agent behavior, see agent evaluation.
Rechunking is not free. Re-embedding and reindexing limits how many attempts you get, so spend them on two or three deliberately different configurations rather than a sweep of nearby token counts. For broader retrieval design questions, see research techniques for better RAG retrieval.
FAQ
What chunk size should I use?
There is no universal number, and any figure quoted without a corpus attached is a guess. Common starting points sit in the 256 to 1,024 token range, with shorter chunks suiting fact lookup in dense reference material and longer ones suiting narrative or procedural documents where reasoning spans paragraphs. Expect the answer to differ across content types inside the same corpus.
What is agentic chunking?
Using a model to decide boundaries instead of a rule. The model reads the document, identifies where topics actually change, and splits there, often attaching a generated summary or heading to make each chunk more retrievable. It handles documents whose structure is inconsistent or absent. The tradeoffs are ingestion cost, latency on large corpora, and boundaries that shift between runs, which complicates comparing two versions of an index.
How much overlap should chunks have?
Enough to survive a bad cut, not enough to bloat the index, with ten to twenty percent of chunk size a common range. Overlap duplicates content across neighbors so a sentence at a boundary appears in both. The cost is near-duplicate chunks competing in the results, which lets heavy overlap crowd the top of the ranking with the same text twice.
Do I need to rechunk when I change embedding models?
Re-embedding is required regardless, since vectors from different models are not comparable. Whether boundaries should change too depends on the new model’s input limit and how it handles short text. If your chunk size was chosen around an older model’s shorter limit, the switch is a good moment to re-run the size comparison while you are reindexing anyway.
How do I know chunking is what is hurting my system?
Look at failures where retrieval returned a topically correct passage and the answer was still wrong. If the retrieved text is cut mid-argument, missing the heading that gave it meaning, or holding a table with no column headers, the problem is upstream of the retriever. If the right document is not coming back at all, the problem is coverage, embeddings, or ranking, and rechunking will not help.
