A text chunk is a smaller segment of a document or conversation used for processing and retrieval. In retrieval-augmented generation (RAG), searching chunks rather than entire documents can locate more precise evidence for a question. Chunks also help keep retrieved information within an LLM’s limited context window, leaving room for the answer. section 7.7.1 section 7.6
The goal is not simply to make every piece small. A useful chunk carries enough context to make sense on its own. Pieces that are too large may dilute relevance or exceed the available context; pieces that are too small may separate information needed to answer a question. Cuts across natural boundaries can also produce incoherent results. section 7.7.1 section 7.7
Choose a splitting method that fits the content and likely queries:
- Fixed length: split by a set number of words or tokens. It is straightforward, but may separate related information. section 7.7.2
- Sliding window: move a fixed-size window through the text, optionally overlapping chunks to retain boundary context. Too much overlap adds redundancy. section 7.7.2 section 7.7.3
- Punctuation or section breaks: use paragraphs, sections, or subsections to preserve natural structure, accepting variable chunk sizes. section 7.7.2
- Adaptive chunking: adjust chunk size to query length and complexity, or retry retrieval at a different granularity when results are unsatisfactory. section 7.7.3
Preparing a PDF for retrieval
For a mostly textual PDF, extract page text with PyPDF2’s PdfReader and extract_text(), then split the resulting text into chunks for indexing. This shortcut ignores images, and PDF text extraction is not always reliable. If tables, images, or document structure matter, consider a document-analysis service such as Azure AI Document Intelligence before chunking the extracted content. section 7.8
Table 7.3 Chunking approaches
|
Chunking approach
|
Description
|
|---|---|
|
Fixed length
|
Divide documents into chunks of a fixed number of words or tokens. This is straightforward but may sometimes split information that ideally should be kept together.
|
|
Sliding window
|
Use a fixed-sized sliding window with or without overlapping data. This can ensure that important boundaries within the text are not missed, but it can also lead to redundancy if there’s significant overlap.
|
|
Punctuation based
|
Divide the text based on punctuation, such as paragraphs or sections. This is less arbitrary than fixed-length chunking and often preserves the semantic integrity of the content. However, it can result in variable chunk sizes.
|
|
Topic or section breaks
|
In structured documents such as Wikipedia articles, natural breaks like sections or subsections can be used to define chunks. This method ensures that the content within a chunk is semantically coherent.
|
|
Adaptive
|
Use algorithms or models that adaptively determine the best way to chunk documents based on their content. This can be more complex but might yield semantically cohesive chunks.
|
Compare splitting approaches before choosing how to balance simple implementation, natural boundaries, and overlap. section 7.7.2
Evaluate the strategy against real queries, not chunk size alone. Account for the source’s length and structure, the LLM and embedding model, query complexity, and how the application uses retrieved results. Unstructured offers preprocessing and auto-chunking capabilities; LangChain and SK support fixed or variable sizes and configurable overlap. In production, chunk quality remains one part of a wider RAG system that must also address cost, governance, privacy, and security. section 7.7.2 section 10.4.2 section 7.6
