A RAG system cannot search a long policy document effectively as one undifferentiated item. During offline indexing, chunking divides documents into smaller units that can be embedded and stored for retrieval. At query time, the system retrieves relevant units and supplies their text to a generator. The choice of boundaries affects what evidence the generator can see. section 1.2 section 4.6
Listing 4.2 Turning a paper into a searchable index.
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_community.vectorstores import FAISS
PAPER = "https://arxiv.org/pdf/2005.11401"
splitter = RecursiveCharacterTextSplitter(
chunk_size=512, chunk_overlap=50
) #A
chunks = PyPDFLoader(PAPER).load_and_split(text_splitter=splitter)
embeddings = HuggingFaceEmbeddings( #B
model_name="sentence-transformers/all-mpnet-base-v2"
)
index = FAISS.from_documents(chunks, embeddings) #C
print(f"{len(chunks)} chunks indexed")The PDF indexing example uses RecursiveCharacterTextSplitter with a chunk size of 512 and an overlap of 50, then embeds the resulting chunks and stores them in FAISS. The split happens before questions are asked; similarity search uses the completed index. section 4.6
Fixed-size splitting is a useful baseline, but its windows can cut across a document’s own divisions. A cheaper structural alternative is to group paragraphs or respect sections. Semantic chunking instead places boundaries where neighboring sentence embeddings diverge; the additional computation is not automatically rewarded with better retrieval. section 4.6 section 10.1.1
When the answer lies between chunks
Suppose an analyst asks how board-meeting risks changed from Q2 to Q3. Splitting each set of minutes into 500-token segments may retrieve isolated mentions of credit risk, supply chains, or a generic disclaimer. None states the comparison. Adding more such snippets does not restore the relationship between the two meetings: the failure is one of specificity and document structure, not simply too few retrieved chunks. section 10.1 section 10.1.1
Choose a boundary strategy against the questions the index must answer:
- Start with fixed-size chunks as a baseline. In a comparison across document- and evidence-retrieval datasets, they performed strongly on natural documents; semantic chunking’s clearest win was on documents assembled from unrelated sources. section 10.1.1
- Test existing document markers before paying for embedding-based boundaries. A separate benchmark found overlapping adjacent paragraphs retrieved better and ran faster than topic-based semantic chunking; its weaker baseline cut text every 100 characters. section 10.1.1
- For thematic questions that require relationships across passages, consider indexing summaries alongside raw chunks. RAPTOR clusters chunks and summarizes them into higher-level nodes, at the cost of additional indexing work. section 10.1 section 10.8
Evaluate retrieval precision and answer quality rather than treating a more elaborate splitter as an upgrade by default. If the needed fact sits in one passage, straightforward chunks may suffice. If the answer depends on narrative or cross-document structure, changing chunk boundaries alone may not supply the missing synthesis. section 4.6 section 10.1 section 10.1.1









