liveBook · Manning

Chunking

As explained in Designing AI Systems

Fixed-size chunking divides text into chunks containing a configured number of tokens with optional overlap between consecutive chunks.

Chunking divides extracted document text into pieces suitable for retrieval. Fixed-size chunking uses a configured token count and can repeat tokens across consecutive chunks. It provides a predictable baseline, but rigid boundaries can split sentences or combine unrelated content. section 5.3.5 section 5.1

In an ingestion pipeline, parsing produces clean text, chunking divides it, and an embedding model converts each chunk into a vector. The vector store retains the chunk text and metadata alongside its vector so search can return relevant passages with source context. section 5.3 section 5.1

Listing 5.11 Fixed-size chunking implementation
class FixedSizeChunking(ChunkingStrategy):
    def chunk(self, document: ExtractedDocument) -> List[Chunk]:
        tokens = tokenize(document.text)    #A
        chunks = []
        start = 0
 
        while start < len(tokens):
            end = min(start + self.chunk_size, len(tokens))
            chunk_tokens = tokens[start:end]
 
            chunk_text = detokenize(chunk_tokens)    #B
            chunks.append(Chunk(
                text=chunk_text,
                start_offset=start,
                end_offset=end
            ))
 
            start += max(1,
                self.chunk_size - self.chunk_overlap)    #C
        return chunks

The fixed-size implementation tokenizes the document, slices token windows, detokenizes each window, and advances by the chunk size minus overlap, with a minimum advance of one token. section 5.3.5

Choosing size and overlap

With a chunk size of 512 tokens and an overlap of 50, consecutive windows begin 462 tokens apart. The shared window can preserve a thought that crosses a boundary, at the cost of duplicated content. It does not make every boundary semantically coherent. section 5.3.5 section 5.2.2

Figure 5.4 Fixed-size chunking with and without overlap. A document is split into three chunks at rigid token boundaries, shown as stacked bars aligned to a token ruler. In the top panel (without overlap), hard cuts at exact token positions split a sentence about defective item returns across two chunks, leaving neither with the complete thought. In the bottom panel (with 50-token overlap), each consecutive pair of chunks shares a window of tokens, shown as the highlighted region where bars overlap. The boundary sentence now appears intact in Chunk 2, making it fully searchable. The overlap costs a small amount of duplicated storage but ensures that information at every boundary is preserved in at least one chunk.

The comparison illustrates how overlap can keep a boundary sentence intact in a retrievable chunk. section 5.3.5

Each chunk carries text, optional heading, offsets, and metadata. Structure-aware chunking can populate the heading; fixed-size chunking leaves it empty. The fixed-size implementation records token positions as offsets, so callers should not assume those values are character positions. section 5.3.5

Listing 5.9 Chunking strategy interface
class ChunkingStrategy(ABC):
    def __init__(
        self,
        chunk_size: int = 512,
        chunk_overlap: int = 50
    ):    #A
        self.chunk_size = chunk_size
        self.chunk_overlap = chunk_overlap
 
    @abstractmethod
    def chunk(
        self, document: ExtractedDocument
    ) -> List[Chunk]:    #B
        """Break a document into chunks for embedding and retrieval."""
        pass

A common strategy interface accepts an extracted document and returns chunks; size and overlap arrive through the constructor. This lets an index use a strategy suited to its content. section 5.3.5

Choose the strategy per index and apply it consistently during ingestion. Fixed-size blocks may merge self-contained FAQ entries, while long technical manuals benefit from boundaries that keep topics separate. Recursive splitting offers another approach: split at paragraph boundaries first, then split oversized pieces at finer boundaries. section 5.2.1 section 5.3.5

How Designing AI Systems covers it

How deep it goes
Implements fixed-size token windows and a pluggable strategy interface, while illustrating recursive splitting and overlap tradeoffs.
Examples
Uses 512-token windows with 50-token overlap, a boundary-spanning return-policy sentence, FAQ entries, and technical manuals.
What you can do after
Build it from scratch as a token-window strategy and configure it per index for ingestion.
How it connects
Connects document parsing, embeddings, vector stores, semantic search, and metadata filtering.

Also explained in

Compare how 10 books explain chunking →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.