Text chunk
A text chunk is a bounded segment of a larger text treated as a unit for processing, storage, or retrieval.
Grokking AI Applications
A contiguous portion of text created by splitting a larger document into smaller units for processing or retrieval.
- How it is used
- Uses chunks as retrievable units of document knowledge for a RAG assistant.
- How deep it goes
- Explains ingestion and retrieval stages and discusses chunk-size, overlap, and result-count tuning.
- Examples
- Uses a restaurant PDF with opening hours, delivery information, and menu prices to build a question-answering assistant.
- What you can do after
- Apply it via Langflow to split document text and tune chunk size and overlap for retrieval.
AI Agents and Applications
A text splitter is a component that divides large text into smaller chunks or document objects.
- How it is used
- Uses chunks to fit processing limits and supply focused context for retrieval and summarization.
- How deep it goes
- Explains size, overlap, structural boundaries, retrieval trade-offs, and parent-child indexing through configured splitters and stores.
- Examples
- Splits travel articles for Chroma search and compares granular HTML sections with coarser character chunks.
- What you can do after
- Apply chunking via LangChain splitters and evaluate retrieval using different chunk sizes and parent-child retrieval.
Generative AI in Action
Chunking is the partitioning of data, especially text, into smaller units for processing, indexing, or retrieval.
- How it is used
- Uses chunking to make enterprise text retrievable within RAG context-window constraints.
- How deep it goes
- Compares boundary and size strategies, selection factors, and adaptive retrieval, with PDF extraction examples.
- Examples
- Illustrates conversational context with a mobile-provider chatbot and PDF extraction with PyPDF2.
- What you can do after
- Evaluate chunking strategies for content and query needs, including overlap, context limits, and PDF extraction constraints.
Spring AI in Action
A document splitter divides a document into smaller chunks suitable for downstream processing.
- How it is used
- Supports RAG ingestion by preparing document chunks for embedding, vector-store storage, and later retrieval as prompt context.
- How deep it goes
- Explains token limits, sentence boundaries, reactive integration, and builder options, but leaves embedding and distance calculations to providers and vector stores.
- Examples
- Uses my-document.txt, uploaded game-rules documents, TokenTextSplitter settings, and a Board Game Buddy loading pipeline.
- What you can do after
- Build it from scratch by reading documents, configuring TokenTextSplitter, and sending the resulting chunks to a vector store.
Fast Python: High performance techniques for large datasets
- How it is used
- Uses chunks to process files in bounded parts and batch work sent between processes.
- How deep it goes
- Develops iterator-based batching and chunked CSV-to-Parquet conversion, including result flattening and cross-chunk type consistency.
- Examples
- Uses a novel word-count benchmark and a taxi CSV-to-Parquet conversion.
- What you can do after
- Apply it via pandas and Arrow to convert CSV data to Parquet incrementally.
Read how Fast Python: High performance techniques for large datasets explains text chunk →
Build AI-Enhanced Web Apps
Chunking is the division of a document or text stream into smaller units for processing or analysis.
- How it is used
- Prepares documents for vector ingestion and summarization within LLM context limits.
- How deep it goes
- Explains recursive separators, size and overlap parameters, document splitting, and downstream processing trade-offs.
- Examples
- Uses PDF and DOCX summarization and a 1,500-character splitter with 200-character overlap.
- What you can do after
- Apply it via LangChain.js to split loaded documents for summarization or vector ingestion.
Designing AI Systems
A document chunk is a segment of an ingested document, with associated source text or metadata, stored as an independently searchable unit.
- How it is used
- Supports passage-level semantic search and grounds model context with retrieved document text.
- How deep it goes
- Defines chunk fields and a strategy interface, then implements fixed-size splitting and illustrates recursive splitting and overlap.
- Examples
- Illustrates overlap with a defective-item return sentence and recursive splitting with paragraph and line boundaries.
- What you can do after
- Builds a chunk model and pluggable fixed-size strategy for an ingestion pipeline.
Essential GraphRAG
Text chunking is the process of splitting source text into smaller pieces for downstream processing and retrieval.
- How it is used
- Improves embedding specificity and retrieval relevance within vector-search and retrieval-augmented generation pipelines.
- How deep it goes
- Defines chunking, compares boundary strategies, and gives sliding-window parameters and Python implementation guidance.
- Examples
- Uses papers, books, whitespace splitting, 500-character windows, 1,000-word limits, and 40-character or word overlaps.
- What you can do after
- Apply it via Python by splitting, overlapping, measuring, and experimentally tuning chunks for retrieval or extraction.
Build an Advanced RAG Application (From Scratch)
A chunking strategy is a method for dividing source text or documents into smaller units for processing or retrieval.
- How it is used
- Supports RAG retrieval by turning long documents into embed-ready units that can return focused evidence for generation.
- How deep it goes
- Explains context-limit motivation and strategy trade-offs, then implements recursive splitting with separator fallbacks and overlap.
- Examples
- Demonstrates hotel dining and spa segments, then splits research-paper abstracts into 657 overlapping chunks.
- What you can do after
- Build it from scratch with a recursive Python splitter that prepares overlapping chunks for embedding and vector search.
Read how Build an Advanced RAG Application (From Scratch) explains text chunk →
MongoDB 8.0 in Action, Third Edition
A text chunk is a bounded, coherent segment of source text, such as a paragraph or section, treated as one unit for processing.
- How it is used
- Supplies retrievable source-document context for a retrieval-augmented generation response.
- How deep it goes
- Explains chunking within a high-level retrieval workflow but does not specify a segmentation algorithm or chunk-size rule.
- Examples
- Uses PDFs and Microsoft Word documents as sources, with paragraphs and sections as possible chunks.
- What you can do after
- Evaluate/understand how coherent source-text segments become retrievable context for a grounded response.
Read how MongoDB 8.0 in Action, Third Edition explains text chunk →
AI-Powered Search
Chunking is the process of dividing a document or other content into smaller sections for separate processing or retrieval.
- How it is used
- Uses chunks to retrieve document passages for dense vector search and retrieval-augmented generation.
- How deep it goes
- Explains chunk boundaries, overlap, indexing alternatives, and granularity trade-offs without specifying a chunking algorithm.
- Examples
- Describes sentence and paragraph chunks, overlapping chunks, and separate chunk documents carrying original document IDs.
- What you can do after
- Evaluate chunk granularity and indexing approaches against context preservation, recall, computational cost, and relevance.
The Art of AI Product Development
Semantic chunking divides text into segments according to meaningful topics or semantic boundaries.
- How it is used
- Uses chunks to retrieve relevant document content for search and response generation.
- How deep it goes
- Compares fixed-size, sentence, paragraph, semantic, and hierarchical chunking, including size trade-offs, without specifying an implementation.
- Examples
- Uses guides, case studies, market reports, financial statistics, and research papers to illustrate chunking choices.
- What you can do after
- Evaluate chunking methods and sizes against document structure and users’ information needs.
Read how The Art of AI Product Development explains text chunk →
Clojure: The Essential Reference
Sequence chunking is an internal optimization that computes a group of sequence elements ahead of their individual demand.
- How it is used
- Uses chunks to improve sequential processing by working through groups of elements rather than strictly one at a time.
- How deep it goes
- Explains buffering and chunk construction, then demonstrates a chunked map; notes the implementation-detail status of the public API.
- Examples
- Builds a three-element chunk, implements a chunked map, and contrasts iterator-seq with enumeration-seq.
- What you can do after
- Build it from scratch with chunk-buffer, chunk-append, chunk, and chunk-cons to process a custom lazy sequence in groups.
Read how Clojure: The Essential Reference explains text chunk →
Effective Conversational AI
A coherent segment of text treated as an individual unit for embedding, indexing, or retrieval.
- How it is used
- Prepares document content for indexing and retrieval in a retrieval-augmented generation pipeline.
- How deep it goes
- Defines chunks as coherent units and mentions paragraph boundaries, but does not specify how to choose chunk sizes.
- Examples
- Uses paragraph-boundary splitting and discusses partitioning large documents while retaining headings, tables, and lists.
- What you can do after
- Evaluate chunking choices as part of preparing documents for more granular retrieval.
Machine Learning Platform Engineering
A text splitter is a component that divides text into smaller chunks according to specified or semantic boundary rules.
- How it is used
- Supports document retrieval by dividing source text into searchable segments before embedding and vector indexing.
- How deep it goes
- Demonstrates configured chunk size, overlap, and separator order, but defers empirical tuning and advanced boundary evaluation.
- Examples
- Uses recursively loaded Markdown, heading-aware separators, 1,000-character chunks, 200-character overlap, FAISS indexing, and a three-result query.
- What you can do after
- Build it from scratch with LangChain by loading documents, splitting them, indexing chunks in FAISS, and testing similarity retrieval.
Read how Machine Learning Platform Engineering explains text chunk →
Data Pipelines with Apache Airflow, Second Edition
A chunk is a smaller piece of source text created by dividing a larger document for retrieval and processing.
- How it is used
- Improves RAG retrieval and source tracing by splitting recipes before embedding and vector-database updates.
- How deep it goes
- Defines chunks, compares oversized and sentence-sized splits, and implements a two-part recipe split without specifying a general sizing algorithm.
- Examples
- Splits each recipe into named ingredient and instruction rows in a pandas DataFrame, then assigns recipe and chunk UUIDs.
- What you can do after
- Build it from scratch with Python and pandas by turning recipe files into source-identifiable chunks for downstream processing.
Read how Data Pipelines with Apache Airflow, Second Edition explains text chunk →
AI Agents in Action
Chunking is the technique of dividing a larger document or text collection into smaller units for processing or retrieval.
- How it is used
- Uses document chunks to supply relevant context for retrieval-augmented generation and agent knowledge.
- How deep it goes
- Demonstrates character- and token-based splitting with overlap and retrieval tests, while deferring a semantic splitting function.
- Examples
- Splits an HTML nursery-rhyme collection and tests retrieval with questions phrased using both similar and different words.
- What you can do after
- Apply chunking via LangChain to a document and test its effect on retrieval.
Microsoft Azure in Action
Chunking is the division of text into smaller segments for processing by a language model or retrieval system.
- How it is used
- Manages text that exceeds what a language model can view at once.
- How deep it goes
- Defines chunking through a suitcase analogy but gives no splitting procedure or configuration.
- Examples
- Uses an oversized book or conversation as the case for splitting text into smaller pieces.
- What you can do after
- Evaluate/understand when splitting text may help keep an LLM request within its input limit.
Build an AI Agent (From Scratch)
Text chunking is the division of source text into smaller segments for storage, retrieval, or processing.
- How it is used
- Makes long documents searchable as smaller units for retrieval-augmented agent context.
- How deep it goes
- Explains size and overlap tradeoffs and implements fixed-length character slicing, without developing other chunking methods.
- Examples
- Uses a 283-character overlap test and a web-results search for quantum computing.
- What you can do after
- Build from scratch a fixed-length chunker and use its output in a vector search pipeline.
Read how Build an AI Agent (From Scratch) explains text chunk →
AI Agents in Action, Second Edition
Semantic chunking is the segmentation of source material into sections that preserve meaningful units of content.
- How it is used
- Supports document ingestion and retrieval by turning source material into searchable context for language-model answers.
- How deep it goes
- Explains token-limited implementation and semantic-coherence guidance, but does not detail advanced boundary algorithms.
- Examples
- Uses a movie script, a 200-token Python-style function, Chroma DB indexing, and retrieved context for question answering.
- What you can do after
- Build it from scratch by splitting documents, embedding chunks, indexing vectors, and grounding responses in retrieved context.
Read how AI Agents in Action, Second Edition explains text chunk →
LLM Customization and Fine-Tuning
A segment of a document that can be indexed or returned as an individual retrieval result.
- How it is used
- Uses chunks as searchable units that supply retrieved context to a RAG prompt.
- How deep it goes
- Defines chunk roles and provides a failure-diagnosis procedure without specifying chunking algorithms.
- Examples
- Illustrates semantic proximity with quarterly revenue, Q3 earnings, and lunch passages, and retrieval with a NumPy matrix and top-k sort.
- What you can do after
- Diagnoses wrong-chunk failures by distinguishing missing or split answers from ranking errors.
Read how LLM Customization and Fine-Tuning explains text chunk →
The Art of Code
Chunking is the grouping of detailed operations into meaningful higher-level units that can be understood as a whole.
- How it is used
- Improves code reading by exposing intent through manageable, named units.
- How deep it goes
- Explains cognitive grouping, demonstrates its memory effect, and applies it through named methods without pursuing implementation internals.
- Examples
- Uses letters versus words, an over-literal walking story, and a createUser method to demonstrate chunking.
- What you can do after
- Apply chunking by extracting coherent operations into clearly named methods that reveal intent and expose omissions.
Building Reliable AI Systems
A segment of text produced by dividing a larger document for processing or retrieval.
- How it is used
- Prepares loaded documents for embedding and precise retrieval within a retrieval-augmented generation indexing pipeline.
- How deep it goes
- Describes splitting loaded data into manageable chunks, including paragraphs or sections, without detailing splitter configuration.
- Examples
- Uses paragraphs or sections as chunk boundaries when dividing loaded documents.
- What you can do after
- Apply text splitters to segment loaded documents into manageable units for embedding and retrieval.
Retrieval Augmented Generation, The Foundational Ideas
A chunk is a smaller contiguous segment of source text created for processing or retrieval.
- How it is used
- Uses contiguous text segments to refine retrieved context and connect raw evidence within graph-based retrieval.
- How deep it goes
- Explains segmentation, query rescoring, filtering, recomposition, and graph bridging, but does not specify chunk-size selection.
- Examples
- Uses an AML policy paragraph or transaction-limit table row and a path through entities, knowledge units, and chunks.
- What you can do after
- Apply chunking to isolate query-relevant evidence and preserve retrieval paths through sparse knowledge graphs.
Read how Retrieval Augmented Generation, The Foundational Ideas explains text chunk →
Also covered in
- Enterprise RAGIn early access (MEAP)
- Logs and Telemetry
- Software Engineering for Data ScientistsIn early access (MEAP)
- Hugging Face in Action
- Just Use Postgres!
- Domain-Specific Small Language Models
- Deep Learning with Python, Third Edition
- API Design Patterns
- Starting Data Analytics with Generative AI and Python
- Natural Language Processing in Action, Second Edition
- Knowledge Graphs and LLMs in Action
- Lead Developer Career Guide
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- similarity searchSimilarity search is a retrieval technique that finds or ranks items by their similarity to a query, often by comparing feature or vector representations under a similarity or distance measure.
- embeddingAn embedding is a numerical vector representation of an item in a feature space, encoding selected properties or relationships for computational tasks such as comparison or retrieval.
- tokenA token is a discrete unit of text, such as a word, subword, punctuation mark, or symbol, produced by tokenization and processed by a language model.
Related concepts
- sliding windowproduced by · A sliding window is a bounded segment or interval that advances across ordered data or time, defining successive portions considered for processing or analysis.
- retrieval-augmented generationuses · Retrieval-augmented generation is a technique that retrieves relevant information from external sources and supplies it as context to a generative model when producing an output.
- embeddingproduces · An embedding is a numerical vector representation of an item in a feature space, encoding selected properties or relationships for computational tasks such as comparison or retrieval.
- chunkingincludes · Chunking is the process of dividing content or data into smaller units for processing, retrieval, or organization.
- RecursiveCharacterTextSplitterimplemented by · RecursiveCharacterTextSplitter is a LangChain text-splitting component that recursively divides text using ordered separators, size constraints, and optional overlap.
Questions about text chunk
What is text chunk?
A text chunk is a bounded segment of a larger text treated as a unit for processing, storage, or retrieval.
Which Manning books explain text chunk?
Grokking AI Applications, AI Agents and Applications, Generative AI in Action, Spring AI in Action, Fast Python: High performance techniques for large datasets, Build AI-Enhanced Web Apps, Designing AI Systems, Essential GraphRAG and 16 more.
What should I know before learning text chunk?
Manning books that teach text chunk usually introduce language model, similarity search, embedding and token first.
What kinds of text chunk are there?
Kinds of text chunk include chunking.
What is text chunk also called?
Text chunk is also called document chunk.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.























