liveBook · Manning

Text chunk

As explained in LLM Customization and Fine-Tuning

A segment of a document that can be indexed or returned as an individual retrieval result.

A text chunk is a segment of a document that can be indexed and returned as an individual retrieval result. In a retrieval-augmented generation (RAG) pipeline, a chunker divides documents into searchable units, often a few hundred tokens each. An embedder represents each chunk as a vector, and an index makes those vectors searchable. section 4.5.2

At query time, the retriever embeds the query and returns the top-k most similar chunks. Prompt assembly adds them as context for the language model. Semantically related chunks—for example, passages about quarterly revenue and Q3 earnings—should be close in the embedding space; cosine similarity is a common way to compare them. section 4.5.2 section 4.5.3

Figure 4.5 Minimal RAG architecture: the ingestion path (chunker, embedder, vector index) runs once at index time; the query path (retriever, prompt assembly, LLM) runs on every request, and both paths meet at the shared vector index.

Follow chunks from document ingestion through the shared vector index to retrieval and prompt assembly. section 4.5.2

When retrieval returns the right document but the wrong chunk, diagnose the failure before changing the search system:

  • Inspect ten production failures and check whether an answer-containing chunk is in the index. If not, the chunker may have dropped or split the answer; revisit the chunking strategy. section 4.5.6
  • If the answer-containing chunk is indexed but ranked too low, investigate hybrid search or reranking instead. section 4.5.6
  • If the correct chunk is retrieved but ignored, strengthen the instruction to use the supplied context and test whether the model follows it. section 4.5.6

How LLM Customization and Fine-Tuning covers it

How deep it goes
Defines chunk roles and provides a failure-diagnosis procedure without specifying chunking algorithms.
Examples
Illustrates semantic proximity with quarterly revenue, Q3 earnings, and lunch passages, and retrieval with a NumPy matrix and top-k sort.
What you can do after
Diagnoses wrong-chunk failures by distinguishing missing or split answers from ranking errors.
How it connects
Connects chunks to vector embeddings, vector indexes, similarity search, prompt assembly, and reranking.

Also explained in

Compare how 24 books explain text chunk →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.