Text chunk
A text chunk is a bounded segment of a larger text or document treated as a unit for processing, storage, or retrieval.
Grokking AI Applications
A contiguous portion of text created by splitting a larger document into smaller units for processing or retrieval.
- How it is used
- Supports RAG retrieval by turning ingested document text into searchable context for LLM answers.
- How deep it goes
- Explains an end-to-end ingestion workflow and tuning trade-offs for chunk size, overlap, and retrieval count.
- Examples
- Uses a restaurant PDF and a visual split into partially overlapping chunks to teach the workflow.
- What you can do after
- Operate/configure it by adjusting chunk size, overlap, and top k to improve RAG responses.
AI Agents and Applications
A text splitter is a component that divides large text into smaller chunks or document objects.
- How it is used
- Prepares large texts for summarization or retrieval by dividing them into pieces that can be processed, embedded, and selected as context.
- How deep it goes
- Works through splitter configuration, document ingestion, size trade-offs, structural splitting, and parent–child retrieval rather than stopping at a definition.
- Examples
- Uses a token-split novel, searchable travel documents, HTML travel-page sections, and parent–child chunks.
- What you can do after
- Apply it via LangChain splitters to create and compare searchable chunks, then retrieve larger parent context when small chunks prove insufficient.
Generative AI in Action
Chunking is the partitioning of data, especially text, into smaller units for processing, indexing, or retrieval.
- How it is used
- Supports RAG by dividing large sources into retrievable units that fit limited LLM context windows.
- How deep it goes
- Compares splitting methods and selection trade-offs, then discusses adaptive retrieval and PDF extraction without deriving an optimal chunk size.
- Examples
- Uses fixed-length and sliding-window comparisons alongside a PyPDF2 PDF-text extraction listing.
- What you can do after
- Apply it via PyPDF2, Unstructured, LangChain, or SK, then assess chunk boundaries and retrieval relevance for the use case.
Spring AI in Action
A document splitter divides a document into smaller chunks suitable for downstream processing.
- How it is used
- Supports RAG ingestion by preparing document chunks for embedding, vector-store storage, and later retrieval as prompt context.
- How deep it goes
- Explains token limits, sentence boundaries, reactive integration, and builder options, but leaves embedding and distance calculations to providers and vector stores.
- Examples
- Uses my-document.txt, uploaded game-rules documents, TokenTextSplitter settings, and a Board Game Buddy loading pipeline.
- What you can do after
- Build it from scratch by reading documents, configuring TokenTextSplitter, and sending the resulting chunks to a vector store.
Fast Python: High performance techniques for large datasets
- How it is used
- Batches words for multiprocessing and reads files in bounded parts when loading the whole input is impractical.
- How deep it goes
- Develops an iterator-based generator and chunked job collection, but does not specify semantic text boundaries or retrieval behavior.
- Examples
- Benchmarks word-count batches on Anna Karenina and converts a CSV file to Parquet one data-frame chunk at a time.
- What you can do after
- Build it from scratch with an iterator-based generator, or apply it via pandas read_csv and ParquetWriter to process bounded batches.
Read how Fast Python: High performance techniques for large datasets explains text chunk →
Build AI-Enhanced Web Apps
Chunking is the division of a document or text stream into smaller units for processing or analysis.
- How it is used
- Prepares long documents for LLM summarization, retrieval, embedding, and vector storage while managing context-window limits.
- How deep it goes
- Explains splitter behavior, character-based size and overlap, and a LangChain.js splitting call; notes that PDF boundaries and large workloads remain difficult.
- Examples
- Shows RecursiveCharacterTextSplitter splitting parsed PDF or DOCX Documents with a 1,500-character size and 200-character overlap.
- What you can do after
- Apply it via LangChain.js text splitters to prepare document chunks for summarization or retrieval.
Essential GraphRAG
Text chunking is the process of splitting source text into smaller pieces for downstream processing and retrieval.
- How it is used
- Improves embedding specificity and retrieval relevance within vector-search and retrieval-augmented generation pipelines.
- How deep it goes
- Defines chunking, compares boundary strategies, and gives sliding-window parameters and Python implementation guidance.
- Examples
- Uses papers, books, whitespace splitting, 500-character windows, 1,000-word limits, and 40-character or word overlaps.
- What you can do after
- Apply it via Python by splitting, overlapping, measuring, and experimentally tuning chunks for retrieval or extraction.
MongoDB 8.0 in Action, Third Edition
A text chunk is a bounded, coherent segment of source text, such as a paragraph or section, treated as one unit for processing.
- How it is used
- Supplies retrievable source-document context for a retrieval-augmented generation response.
- How deep it goes
- Explains chunking within a high-level retrieval workflow but does not specify a segmentation algorithm or chunk-size rule.
- Examples
- Uses PDFs and Microsoft Word documents as sources, with paragraphs and sections as possible chunks.
- What you can do after
- Evaluate/understand how coherent source-text segments become retrievable context for a grounded response.
Read how MongoDB 8.0 in Action, Third Edition explains text chunk →
AI-Powered Search
Chunking is the process of dividing a document or other content into smaller sections for separate processing or retrieval.
- How it is used
- Supports passage-level matching in dense vector search and retrieval of supporting sections for RAG prompts.
- How deep it goes
- Explains boundary choices, overlap, two indexing layouts, and trade-offs in context, recall, cost, and ranking; provides no chunking algorithm.
- Examples
- Uses sentence and paragraph boundaries, overlapping chunks, and a document split into ten independently embedded chunks.
- What you can do after
- Operate/configure it by choosing chunk granularity and an indexing layout while weighing context preservation, recall, and search cost.
The Art of AI Product Development
Semantic chunking divides text into segments according to meaningful topics or semantic boundaries.
- How it is used
- Supplies retrievable document segments for search and context for retrieval-augmented generation.
- How deep it goes
- Compares fixed-size, sentence, paragraph, semantic, and hierarchical splitting, then weighs chunk size against user needs without detailing splitting algorithms.
- Examples
- Uses how-to guides, market reports, annual reports, and financial statistics to distinguish useful boundaries and sizes.
- What you can do after
- Operate/configure it by choosing chunking methods and sizes for document types and user needs, then evaluating retrieval and answer quality.
Read how The Art of AI Product Development explains text chunk →
Clojure: The Essential Reference
Sequence chunking is an internal optimization that computes a group of sequence elements ahead of their individual demand.
- How it is used
- Uses chunks to improve sequential processing by working through groups of elements rather than strictly one at a time.
- How deep it goes
- Explains buffering and chunk construction, then demonstrates a chunked map; notes the implementation-detail status of the public API.
- Examples
- Builds a three-element chunk, implements a chunked map, and contrasts iterator-seq with enumeration-seq.
- What you can do after
- Build it from scratch with chunk-buffer, chunk-append, chunk, and chunk-cons to process a custom lazy sequence in groups.
Read how Clojure: The Essential Reference explains text chunk →
Effective Conversational AI
A coherent segment of text treated as an individual unit for embedding, indexing, or retrieval.
- How it is used
- Prepares document content for indexing and retrieval in a retrieval-augmented generation pipeline.
- How deep it goes
- Defines chunks as coherent units and mentions paragraph boundaries, but does not specify how to choose chunk sizes.
- Examples
- Uses paragraph-boundary splitting and discusses partitioning large documents while retaining headings, tables, and lists.
- What you can do after
- Evaluate chunking choices as part of preparing documents for more granular retrieval.
Machine Learning Platform Engineering
A text splitter is a component that divides text into smaller chunks according to specified or semantic boundary rules.
- How it is used
- Supports document retrieval by dividing source text into searchable segments before embedding and vector indexing.
- How deep it goes
- Demonstrates configured chunk size, overlap, and separator order, but defers empirical tuning and advanced boundary evaluation.
- Examples
- Uses recursively loaded Markdown, heading-aware separators, 1,000-character chunks, 200-character overlap, FAISS indexing, and a three-result query.
- What you can do after
- Build it from scratch with LangChain by loading documents, splitting them, indexing chunks in FAISS, and testing similarity retrieval.
Read how Machine Learning Platform Engineering explains text chunk →
Data Pipelines with Apache Airflow, Second Edition
A chunk is a smaller piece of source text created by dividing a larger document for retrieval and processing.
- How it is used
- Improves RAG retrieval and source tracing by splitting recipes before embedding and vector-database updates.
- How deep it goes
- Defines chunks, compares oversized and sentence-sized splits, and implements a two-part recipe split without specifying a general sizing algorithm.
- Examples
- Splits each recipe into named ingredient and instruction rows in a pandas DataFrame, then assigns recipe and chunk UUIDs.
- What you can do after
- Build it from scratch with Python and pandas by turning recipe files into source-identifiable chunks for downstream processing.
Read how Data Pipelines with Apache Airflow, Second Edition explains text chunk →
Microsoft Azure in Action
Chunking is the division of text into smaller segments for processing by a language model or retrieval system.
- How it is used
- Manages text that exceeds what a language model can view at once.
- How deep it goes
- Defines chunking through a suitcase analogy but gives no splitting procedure or configuration.
- Examples
- Uses an oversized book or conversation as the case for splitting text into smaller pieces.
- What you can do after
- Evaluate/understand when splitting text may help keep an LLM request within its input limit.
Build an AI Agent (From Scratch)
Text chunking is the division of source text into smaller segments for storage, retrieval, or processing.
- How it is used
- Turns long documents into searchable units so an agent can retrieve relevant passages instead of placing entire documents in context.
- How deep it goes
- Implements fixed-length slicing with overlap and explains size trade-offs; leaves optimal size dependent on document characteristics.
- Examples
- Tests overlap on 283 characters and retrieves quantum-related passages from chunked web-search results.
- What you can do after
- Build it from scratch with a Python chunking function, then apply it to embedding and searching web-page passages.
Read how Build an AI Agent (From Scratch) explains text chunk →
AI Agents in Action, Second Edition
Semantic chunking is the segmentation of source material into sections that preserve meaningful units of content.
- How it is used
- Supports document ingestion and retrieval by turning source material into searchable context for language-model answers.
- How deep it goes
- Explains token-limited implementation and semantic-coherence guidance, but does not detail advanced boundary algorithms.
- Examples
- Uses a movie script, a 200-token Python-style function, Chroma DB indexing, and retrieved context for question answering.
- What you can do after
- Build it from scratch by splitting documents, embedding chunks, indexing vectors, and grounding responses in retrieved context.
Read how AI Agents in Action, Second Edition explains text chunk →
The Art of Code
Chunking is the grouping of detailed operations into meaningful higher-level units that can be understood as a whole.
- How it is used
- Improves code reading by exposing intent through manageable, named units.
- How deep it goes
- Explains cognitive grouping, demonstrates its memory effect, and applies it through named methods without pursuing implementation internals.
- Examples
- Uses letters versus words, an over-literal walking story, and a createUser method to demonstrate chunking.
- What you can do after
- Apply chunking by extracting coherent operations into clearly named methods that reveal intent and expose omissions.
Building Reliable AI Systems
A segment of text produced by dividing a larger document for processing or retrieval.
- How it is used
- Prepares loaded documents for embedding and precise retrieval within a retrieval-augmented generation indexing pipeline.
- How deep it goes
- Describes splitting loaded data into manageable chunks, including paragraphs or sections, without detailing splitter configuration.
- Examples
- Uses paragraphs or sections as chunk boundaries when dividing loaded documents.
- What you can do after
- Apply text splitters to segment loaded documents into manageable units for embedding and retrieval.
Related concepts
- sliding windowproduced by · A sliding window is a bounded region over an ordered sequence or time range that advances, updating which elements fall within it.
- retrieval-augmented generationuses · Retrieval-augmented generation is a technique that retrieves relevant information from external sources and supplies it as context to a generative model when producing an output.
- embeddingproduces · An embedding is a numerical vector representation of an item that encodes features or relationships useful for computational tasks.
- chunkingincludes · Chunking is the process of dividing content or data into smaller units for processing, retrieval, or organization.
- RecursiveCharacterTextSplitterimplemented by · RecursiveCharacterTextSplitter is a LangChain text-splitting component that recursively divides text using ordered separators, size constraints, and optional overlap.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.