In the entire RAG pipeline, chunking offers the highest return on investment: without swapping models or adding components, tweaking a few parameters can significantly boost retrieval accuracy. Yet, it is also the step most often decided by guesswork—many systems simply stick with "whatever the framework defaults to," leaving those settings untouched from launch until decommissioning.
This article clarifies three key points: why chunking methods directly determine answer quality, which of the four mainstream strategies suits different document types, and a ready-to-use default configuration along with an optimization workflow you can follow immediately.
The essence of chunking: breaking down documents into units that are both retrievable and capable of independently answering questions.
Why Chunking Matters So Much
Retrieval operates at the chunk level, meaning each chunk must simultaneously satisfy two conflicting requirements:
- Small enough: Content should be focused so that its Embedding vector accurately captures its semantics, ensuring precise retrieval;
- Large enough: Once retrieved, it must contain complete information on its own, giving the model sufficient context to generate an answer.
If chunks are too fragmented, a hit might yield only half a sentence, prompting the model to hallucinate and hallucination ensues; if they’re too large, a single chunk may blend three or four topics, turning its vector into an "average semantic" that touches on everything but matches nothing precisely. Optimizing chunk size is about finding the balance between these extremes—a balance point that varies by document type. This explains why there’s no universal “best” chunk size applicable everywhere.
Four Mainstream Chunking Strategies
1. Fixed-Size Chunking (Fixed-size)
This approach performs hard cuts at fixed token or character intervals, typically with a configured overlap. It is the simplest to implement and offers the fastest speed but risks splitting sentences in half or cutting through tables mid-stream. Consequently, it is suitable only for rapid prototyping or purely textual content with highly uniform structure.
2. Recursive Chunking (Recursive)
The default strategy in mainstream frameworks: recursively split text by descending priority of delimiters (paragraphs → newlines → periods → spaces), aiming to break at natural boundaries and degrading only when chunks exceed length limits. Both LangChain's RecursiveCharacterTextSplitter and LlamaIndex's SentenceSplitter follow this approach. It offers the best general applicability, making it a reasonable starting point for most scenarios.
3. Structure-Aware Chunking (Structure-aware)
Chunk based on the document’s inherent structure: by heading levels in Markdown, DOM elements in HTML, functions or classes in code, and clauses in contracts. Each chunk naturally corresponds to a complete semantic unit, allowing you to embed the path "Document Name > Chapter > Subsection" into metadata or prepend it to the chunk itself. For documents that already possess structure, this strategy yields the highest possible performance, though it requires writing custom parsing logic for each document type.
4. Semantic Chunking (Semantic)
Compute embeddings sentence by sentence and split where the semantic meaning shifts abruptly between adjacent sentences. This approach suits long-form texts lacking structural formatting, such as interview transcripts, subtitles, or plain-text novels, enabling the creation of thematically coherent chunks. The downsides are high ingestion costs (requiring vector calculation for every sentence) and unstable splitting results; therefore, it is generally reserved for cases where the first three strategies yield unsatisfactory outcomes.
| Strategy | Implementation Cost | Performance Ceiling | Suitable Documents |
|---|---|---|---|
| Fixed Length | Very Low | Low | Prototype Validation |
| Recursive Splitting | Low | Medium | General Default |
| Structured Splitting | Medium | High | Markdown/HTML/Code/Contracts |
| Semantic Chunking | High | Medium-High | Unstructured Long-Form Texts |
Practical Comparison of Chunk Sizes: How to Choose the Right Size
We tested several common chunk sizes using a consistent set of Chinese documents (a mix of policy manuals, technical guides, and FAQs totaling approximately 800 pages) under identical retrieval configurations (BGE-family embeddings with Top-5 recall). Our findings align with mainstream community experience:
| Chunk Size (tokens) | Retrieval Hit Performance | Typical Issues |
|---|---|---|
| 128–256 | Excellent for precise question matching | Answers spanning multiple chunks get cut off, requiring overlap and multi-retrieval to compensate. |
| 384–512 | Most stable overall | A few long-process questions still require extended context. |
| 1024 | Hit rate begins to decline | Vector semantics become "averaged," leading to an increase in mixed-topic chunks. |
| 2048+ | Significant drop-off | Multiple topics per chunk degrade both retrieval accuracy and answer focus. |
Here are a few actionable takeaways:
- Start with a default of 512 tokens and 10%–20% overlap. Use smaller sizes (around 256) for FAQs and short clauses, and larger ones (512–768) for workflows and tutorials.
- More overlap is not always better: Exceeding 30% makes adjacent chunks highly similar, causing the same content to repeat in retrieval results and crowding out candidate slots.
- When estimating by character count for Chinese text, remember that 1 token roughly equals 1–1.5 Chinese characters; thus, 512 tokens correspond approximately to 400–700 characters. Do not directly apply English-based heuristics.
Advanced Techniques: Decoupling Retrieval and Generation Units
In the later stages of tuning, the most effective strategy is to stop treating "the block used for retrieval" as identical to "the block shown to the model":
- Small-to-Big / Parent-Child Chunks: Use small chunks (e.g., 256 tokens) for retrieval to ensure precision. Once a match is found, return its parent chunk (e.g., 1024 tokens or an entire section) to the model, balancing both needs.
SentenceWindowNodeParserin LlamaIndex andParentDocumentRetrieverin LangChain offer ready-made implementations of this approach. - Contextual Chunking: Before indexing, use a model to generate a prefix for each chunk stating "which document it comes from and what it discusses," then vectorize the result. This aligns with Anthropic's Contextual Retrieval concept, significantly improving performance on documents heavy in pronouns or dependent on context. The trade-off is an additional model call during indexing (though costs can be minimized by leveraging Prompt Caching).
- Separate Handling for Tables and Code: Convert tables to Markdown or expand them row-by-row into sentences; split code by function while preserving file paths. Mixing these content types with standard text chunkers almost inevitably leads to issues.
Parent-child retrieval: Small chunks handle being found, large chunks ensure the message is complete.
A Tuning Workflow You Can Copy Directly
- Build an Evaluation Set: Select 50 real-world queries and annotate which document segments each should retrieve. Without this step, everything that follows is guesswork.
- Run a Baseline: Use recursive chunking with 512 tokens and 15% overlap; record the retrieval hit rate (whether correct segments appear in the Top-5).
- Adjust One Variable at a Time: Change only the chunk size (run separate tests for 256, 512, and 768), then select the configuration yielding the highest hit rate.
- Switch Strategies: If documents have structure (headings, clauses, code), switch to structured chunking and run another round; performance usually improves further.
- Implement Parent-Child Chunks: When you see high retrieval rates but frequent "incomplete answers," add Small-to-Big logic.
- Solidify and Monitor: Embed the configuration into your codebase and documentation. Re-run the evaluation set whenever you change embedding models or significantly alter the corpus.
Note: Changing chunking parameters necessitates a full index rebuild. For large corpora, evaluate both reconstruction time and vectorization costs. In production systems, run old and new indexes in parallel with gradual traffic shifting to avoid retrieval quality fluctuations during the rebuild window.
Change only one variable at a time using the same evaluation set; you can optimize chunk size within a single day.
Target Audience and Alternatives
This guide is for engineers and technical product managers building or optimizing RAG knowledge bases. If your scenario involves a total document volume of less than tens of thousands of words, you can skip chunking entirely and feed the full text directly into the context window. If you are using platforms like Dify, FastGPT, or Coze, the sizing heuristics in this article still apply—their "chunk settings" and "parent-child chunking" features are simply encapsulated versions of the strategies discussed above. The only scenarios where chunking cannot be bypassed involve large-scale corpora that require frequent updates and strict cost control in self-hosted deployments.
Frequently Asked Questions
Q: Should I tune chunking or retrieval methods first? A: Tune chunking before retrieval. Chunking is a data quality issue; if the data is poorly segmented, Hybrid Search and Rerank are merely applying patches to bad data. For the relationship between these three components, see "Why Does RAG Give Inaccurate Answers? 10 Engineering Causes and Solutions".
Q: If I switch embedding models, do I need to retune the chunk size? A: It is recommended to re-run your evaluation set at least once. Different models have different optimal input lengths (most embedding models perform best with inputs of 256–512 tokens), though a default value of 512 usually remains safe.
Q: What if my knowledge base contains both FAQs and long manuals? A: Chunk by document type separately (e.g., one question-and-answer pair per chunk for FAQs, structured chapter-based chunks for manuals) and tag the documents with type metadata upon ingestion. Do not use a single set of parameters to process all documents just for convenience.
Summary
There is no silver bullet for chunking, but there is a clear methodology: start with recursive chunking at 512 tokens by default; apply structured chunking to documents that have structure; switch to parent-child chunks if answers are incomplete. Let an evaluation set of 50 examples speak for every change you make. Solidifying this step often improves RAG accuracy more than upgrading the model ever could.