Chunking
Chunking is the process of dividing content or data into smaller units for processing, retrieval, or organization.
Generative AI in Action
Fixed-length chunking divides text into chunks according to a predetermined character, word, or token length rather than linguistic boundaries.
- How it is used
- Prepares large corpora for RAG retrieval when predictable sizing, performance, and scalability matter more than linguistic boundaries.
- How deep it goes
- Defines fixed-size splitting, demonstrates lost context, and compares selection factors without detailing a complete implementation.
- Examples
- Uses the UK Constitution, fixed-size figures, and comparisons with textwrap, NLTK, spaCy, and NLP-based splitting.
- What you can do after
- Apply it via a fixed-size splitter, then evaluate retrieval context, performance, resource use, and corpus structure against alternatives.
The Art of AI Product Development
Fixed-size chunking is the division of documents into segments containing a predetermined number of tokens or other units.
- How it is used
- Prepares varied-length documents for semantic embedding and search by splitting them into segments of similar length.
- How deep it goes
- Explains the length trade-off and gives a 300-token example, but does not detail chunk-boundary strategies.
- Examples
- Uses lengthy notes and reports split into 300-token chunks before MiniLM embedding and storage in Weaviate.
- What you can do after
- Apply it by splitting documents into similarly sized segments before embedding them for semantic search.
Read how The Art of AI Product Development explains chunking →
Building Reliable AI Systems
A method for dividing source material into segments for processing or retrieval.
- How it is used
- Supports embedding and retrieval by dividing documents into pieces whose boundaries affect what a RAG system can find.
- How deep it goes
- Explains boundary trade-offs and gives content-type rules plus Python-style dispatch and sliding-window implementations; leaves specialized parsers as function calls.
- Examples
- Uses a CFRA eligibility question, document-type chunking rules, and Python-style sliding-window code.
- What you can do after
- Apply it via content-type chunkers and overlapping windows, then label chunks with source metadata for retrieval.
Financial AI in Practice
Chunking divides documents or other content into smaller segments for processing or retrieval.
- How it is used
- Improves retrieval in RAG pipelines by balancing specific matches against enough surrounding context for grounded answers.
- How deep it goes
- Explains the small-versus-large chunk tradeoff and parent-document retrieval without detailing a full implementation.
- Examples
- Compares 50,000-word Form 10-K filings with 250-character summaries and 80-character headlines that remain intact.
- What you can do after
- Evaluate whether to split content, retain it intact, or retrieve parent documents after searching smaller chunks.
Design for Developers
A technique for dividing content or a task into smaller, manageable units or sequential steps.
- How it is used
- Uses smaller steps to make long mobile forms manageable and reduce user overload.
- How deep it goes
- Explains sequential screens, progress cues, and focused error correction, but does not specify implementation code.
- Examples
- Uses payment, shipping address, and shipping options as separate checkout steps.
- What you can do after
- Apply chunking by splitting a long mobile form into focused screens with clear movement between steps.
Related concepts
- information retrievaluses · Information retrieval is the process of locating and selecting relevant information from an available collection in response to a query or information need.
- text chunkkind of · A text chunk is a bounded segment of a larger text treated as a unit for processing, storage, or retrieval.
- RecursiveCharacterTextSplitterincludes · RecursiveCharacterTextSplitter is a LangChain text-splitting component that recursively divides text using ordered separators, size constraints, and optional overlap.
- cognitive loadaddresses · Cognitive load is the mental effort required to process information, retain it in working memory, or perform a task.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.