liveBook · Manning

Chunking

As explained in Building Reliable AI Systems

A method for dividing source material into segments for processing or retrieval.

Chunking divides documents into smaller pieces for embedding and retrieval. Granularity affects what a retrieval-augmented generation (RAG) system can find: small chunks can separate necessary context, while large chunks can dilute relevance with unrelated content. section 4.4.2 section 4.7

Policy eligibility

A policy passage names a leave program and states its eligibility requirements. At 100–200 tokens, a split may separate those facts, so a query retrieves the policy description but misses the criteria. A chunk over 500 tokens can keep them together, but unrelated policies may dilute its relevance score. section 4.4.2

Content-aware chunking uses document structure to keep related information together rather than applying one split everywhere. section 4.4.2

Choose boundaries to suit the content:

  • For legal documents, identify section headers or numbered clauses and preserve complete concepts. section 4.4.2
  • For narrative content, split on paragraph breaks and use a minimum length to limit fragmentation. section 4.4.2
  • For code documentation, detect code fences, function signatures, and other delimiters so examples remain paired with explanations. section 4.4.2

When specialized parsing does not apply, a sliding window can overlap adjacent chunks. Label chunks with metadata such as section headings, page numbers, filenames, and publication dates to retain structured context and support filtering. section 4.4.2 section 4.7

def sliding_window_chunks(text, chunk_size=500, overlap=100):
    """Create overlapping chunks using a sliding window"""
    tokens = text.split()
    chunks = []

    for i in range(0, len(tokens), chunk_size - overlap):
        chunk = tokens[i:i + chunk_size]    #1
        if chunk:    #2
            chunks.append(" ".join(chunk))

    return chunks

This Python fallback splits on whitespace and advances by chunk size minus overlap; its example uses a 500-word window and 100-word overlap. section 4.4.2

How Building Reliable AI Systems covers it

How deep it goes
Explains size trade-offs and content-specific boundaries, with a sliding-window implementation.
Examples
Uses a leave-policy eligibility passage and legal, narrative, and code documentation splitting strategies.
What you can do after
Evaluate chunking choices for context retention and retrieval relevance, and apply content-aware or overlapping splits.
How it connects
Connects chunking to embeddings, RAG, relevance scoring, and metadata filtering.

Also explained in

Compare how 10 books explain chunking →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Reliable AI Systems. The section links point to the parts of the book it draws on.