A text chunk is a smaller portion of a larger document. Instead of asking a system to handle an entire PDF, web page, or other file as one enormous piece, document ingestion first extracts plain text and splits it into chunks. Each chunk can then be represented as an embedding and stored in a vector database, ready for later search. section 4.2 section 4.5
Chunks make retrieval practical in a retrieval-augmented generation (RAG) flow. When someone asks a question, the system searches for the few chunks most relevant to that query rather than treating every source document as equally useful. Those retrieved pieces become extra context for the LLM’s answer. section 4.1 section 4.5
A typical chunking pipeline has five stages:
- Read a source file and extract its text. section 4.2
- Split the raw text into smaller chunks, optionally with partial overlap. section 4.2
- Turn every chunk into a numeric embedding with an embedding model. section 4.2
- Store the embeddings in a vector database. section 4.2
- At runtime, retrieve the most relevant chunks and pass them to the LLM with the user query. section 4.2
Figure 4.13 The raw text is split into chunks, partially overlapping each other.
Overlapping chunks preserve some neighboring context at chunk boundaries instead of making every piece completely separate. section 4.2
Chunk boundaries and sizes are design choices, not mere preprocessing details. Very small chunks may leave out information needed for a correct answer. Very large chunks can make the supplied context too large, degrade LLM performance, and produce embeddings that do not represent all of the chunk’s content effectively. Chunk overlap can be adjusted alongside chunk size to find a useful balance. section 4.3
Retrieval settings interact with chunking. A similarity search returns a chosen number of relevant chunks, often called top k. Retrieving too many can confuse the LLM; retrieving too few can omit a critical piece. Test chunk size, overlap, and top k together when answers are weak or incomplete. section 4.3
Restaurant assistant
A restaurant PDF can contain working hours, delivery information, menu items, and prices. Split its extracted text into chunks, embed and store them, then retrieve the relevant chunks when a customer asks a question. The LLM can answer using the restaurant information retrieved at runtime; rerun ingestion when the PDF changes. section 4.2 section 4.5
A chunk is therefore both a unit of storage and a unit of retrieved context. Good chunking helps a RAG system supply the right evidence to an LLM, but it does not guarantee success: retrieval quality depends on how well the chunks preserve the information a question needs and on how many are selected. section 4.1 section 4.3
