liveBook · Manning

Chunking

As explained in Retrieval Augmented Generation, The Foundational Ideas

Chunking is the division of source documents into smaller units for processing and retrieval.

A RAG system cannot search a long policy document effectively as one undifferentiated item. During offline indexing, chunking divides documents into smaller units that can be embedded and stored for retrieval. At query time, the system retrieves relevant units and supplies their text to a generator. The choice of boundaries affects what evidence the generator can see. section 1.2 section 4.6

Listing 4.2 Turning a paper into a searchable index.
from langchain_community.document_loaders import PyPDFLoader 
from langchain_text_splitters import RecursiveCharacterTextSplitter 
from langchain_huggingface import HuggingFaceEmbeddings 
from langchain_community.vectorstores import FAISS
PAPER = "https://arxiv.org/pdf/2005.11401" 
splitter = RecursiveCharacterTextSplitter(
  chunk_size=512, chunk_overlap=50
)  #A
chunks = PyPDFLoader(PAPER).load_and_split(text_splitter=splitter) 
embeddings = HuggingFaceEmbeddings(  #B
  model_name="sentence-transformers/all-mpnet-base-v2"
)
index = FAISS.from_documents(chunks, embeddings)  #C
print(f"{len(chunks)} chunks indexed")

The PDF indexing example uses RecursiveCharacterTextSplitter with a chunk size of 512 and an overlap of 50, then embeds the resulting chunks and stores them in FAISS. The split happens before questions are asked; similarity search uses the completed index. section 4.6

Fixed-size splitting is a useful baseline, but its windows can cut across a document’s own divisions. A cheaper structural alternative is to group paragraphs or respect sections. Semantic chunking instead places boundaries where neighboring sentence embeddings diverge; the additional computation is not automatically rewarded with better retrieval. section 4.6 section 10.1.1

When the answer lies between chunks

Suppose an analyst asks how board-meeting risks changed from Q2 to Q3. Splitting each set of minutes into 500-token segments may retrieve isolated mentions of credit risk, supply chains, or a generic disclaimer. None states the comparison. Adding more such snippets does not restore the relationship between the two meetings: the failure is one of specificity and document structure, not simply too few retrieved chunks. section 10.1 section 10.1.1

Choose a boundary strategy against the questions the index must answer:

  • Start with fixed-size chunks as a baseline. In a comparison across document- and evidence-retrieval datasets, they performed strongly on natural documents; semantic chunking’s clearest win was on documents assembled from unrelated sources. section 10.1.1
  • Test existing document markers before paying for embedding-based boundaries. A separate benchmark found overlapping adjacent paragraphs retrieved better and ran faster than topic-based semantic chunking; its weaker baseline cut text every 100 characters. section 10.1.1
  • For thematic questions that require relationships across passages, consider indexing summaries alongside raw chunks. RAPTOR clusters chunks and summarizes them into higher-level nodes, at the cost of additional indexing work. section 10.1 section 10.8

Evaluate retrieval precision and answer quality rather than treating a more elaborate splitter as an upgrade by default. If the needed fact sits in one passage, straightforward chunks may suffice. If the answer depends on narrative or cross-document structure, changing chunk boundaries alone may not supply the missing synthesis. section 4.6 section 10.1 section 10.1.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Works through a PDF indexing configuration and compares boundary strategies, then identifies where flat chunks fail thematic retrieval.
Examples
Uses a chunked PDF index and a Q2-versus-Q3 board-minutes question whose answer spans separate passages.
What it assumes
Introduces chunk-and-embed indexing through a PDF example; reserves clustering and dimensionality-reduction mathematics for hierarchical retrieval.
What you can do after
Evaluate/understand it by testing fixed-size and structural boundaries against retrieval quality before adopting costlier chunking or hierarchical indexing.
How it connects
Connects chunk boundaries to embeddings, vector search, semantic chunking, paragraph grouping, and RAPTOR’s summary-node retrieval.

Also explained in

Compare how 10 books explain chunking →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.