liveBook · Manning

Text chunk

As explained in Designing AI Systems

A document chunk is a segment of an ingested document, with associated source text or metadata, stored as an independently searchable unit.

A text chunk is a passage extracted from a document and stored as an independently searchable unit, with source context and metadata. Chunking lets semantic search compare a question with focused passages rather than an entire document. Chunks that are too large dilute relevant information; chunks that are too small lose context. section 5.1 section 5.3.5

An ingestion pipeline detects the file format, extracts text and structure, captures metadata, then chunks and embeds the text. Each vector is stored with its source text and metadata. At query time, similarity search returns matching chunks for use as grounding information. section 5.3.2 section 5.1

Figure 5.1 The retrieval pipeline from document to search result. A source document enters the ingestion pipeline, where it is parsed into clean text and then broken into semantically coherent chunks. Each chunk is passed through an embedding model to produce a dense vector representation. The vectors, along with the original chunk text and metadata, are stored in a vector database. At query time, the user's question follows the same embedding process to produce a query vector, which is compared against stored vectors using similarity search. The top matching chunks are returned with their source attribution, relevance scores, and metadata, ready to be injected into the model's context as grounding information.

Follow a chunk from document parsing through embedding, storage, and retrieval. section 5.1

A chunk model should preserve the text to embed and return, an optional heading, source offsets, and per-chunk metadata. Headings and offsets help trace a result back to its source; metadata can support finer-grained filtering. Structure-aware chunking can populate headings, while fixed-size chunking leaves them empty. section 5.3.5

Listing 5.8 Chunk dataclass
@dataclass
class Chunk:
    text: str    #A
    heading: Optional[str] = None    #B
    start_offset: int = 0    #C
    end_offset: int = 0    #C
    metadata: Dict[str, str] = field(
        default_factory=dict
    )    #D

Inspect the chunk dataclass and its source-context fields. section 5.3.5

A common strategy interface accepts an extracted document and returns chunks. Configure chunk size, overlap, and strategy on the index so that the same choices apply to each ingested document. Different strategies can then share the same downstream embedding and storage pipeline. section 5.3.5

Two ways to split text illustrate the tradeoffs:

  • Fixed-size chunking uses token boundaries. Overlap repeats a window between neighboring chunks, helping preserve boundary-spanning information at the cost of duplicated storage. section 5.3.5
  • Recursive splitting tries structural separators in order: pieces that fit become chunks, while oversized pieces are split again using a finer separator. section 5.3.5
Listing 5.11 Fixed-size chunking implementation
class FixedSizeChunking(ChunkingStrategy):
    def chunk(self, document: ExtractedDocument) -> List[Chunk]:
        tokens = tokenize(document.text)    #A
        chunks = []
        start = 0
 
        while start < len(tokens):
            end = min(start + self.chunk_size, len(tokens))
            chunk_tokens = tokens[start:end]
 
            chunk_text = detokenize(chunk_tokens)    #B
            chunks.append(Chunk(
                text=chunk_text,
                start_offset=start,
                end_offset=end
            ))
 
            start += max(1,
                self.chunk_size - self.chunk_overlap)    #C
        return chunks

Inspect a fixed-size implementation that tokenizes text, creates chunks, and advances by size minus overlap. section 5.3.5

A boundary-spanning return policy

A rigid token boundary can split a sentence about defective-item returns so that neither chunk contains the complete thought. A 50-token overlap can place the complete sentence in a neighboring chunk, making that passage searchable. section 5.3.5

The vector store associates each chunk and embedding with an index and document ID. This scope supports deleting all chunks for an updated document or a removed index. Search results carry the chunk and document IDs, text, score, and metadata; production deletion should check caller authorization. section 5.4.1

How Designing AI Systems covers it

How deep it goes
Defines chunk fields and a strategy interface, then implements fixed-size splitting and illustrates recursive splitting and overlap.
Examples
Illustrates overlap with a defective-item return sentence and recursive splitting with paragraph and line boundaries.
What you can do after
Builds a chunk model and pluggable fixed-size strategy for an ingestion pipeline.
How it connects
Connects document parsing, metadata filtering, embeddings, vector stores, and similarity search.

Also explained in

Compare how 24 books explain text chunk →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.