A content-generation app can retrieve the right report and still miss the detail a user needs. A text chunk is a segment of a document that the search system can retrieve. If every document is split into fixed 200-token pieces, a useful topic may not fit neatly within a piece. Chunking is therefore a product decision as well as a preprocessing choice: it affects which information reaches the user. section 7.2.4
Choose boundaries and sizes to match the documents and the questions users ask:
- For short how-to guides or case studies, split by sentence or paragraph when each part stands alone. section 7.2.4
- For industry analysis or market reports, use semantic chunking to make each segment correspond to a distinct topic. section 7.2.4
- For annual reports or research papers, divide into chapters or sections, then into smaller chunks, so users can reach either an overview or a detail. section 7.2.4
- Favor shorter chunks for specific facts, such as financial statistics; consider longer ones when drafting needs broader context. section 7.2.4
In retrieval-augmented generation, the top retrieved chunks become context in the language-model prompt. Multiple chunks make prompts longer, so retrieval choices also affect input-token cost and how well the model uses the provided context. Chunking alone does not guarantee a reliable answer: test context relevance, groundedness, and answer relevance, and adjust retrieval alongside other search settings. section 7.3.1 section 7.4
