Chunking divides extracted document text into pieces suitable for retrieval. Fixed-size chunking uses a configured token count and can repeat tokens across consecutive chunks. It provides a predictable baseline, but rigid boundaries can split sentences or combine unrelated content. section 5.3.5 section 5.1
In an ingestion pipeline, parsing produces clean text, chunking divides it, and an embedding model converts each chunk into a vector. The vector store retains the chunk text and metadata alongside its vector so search can return relevant passages with source context. section 5.3 section 5.1
Listing 5.11 Fixed-size chunking implementation
class FixedSizeChunking(ChunkingStrategy):
def chunk(self, document: ExtractedDocument) -> List[Chunk]:
tokens = tokenize(document.text) #A
chunks = []
start = 0
while start < len(tokens):
end = min(start + self.chunk_size, len(tokens))
chunk_tokens = tokens[start:end]
chunk_text = detokenize(chunk_tokens) #B
chunks.append(Chunk(
text=chunk_text,
start_offset=start,
end_offset=end
))
start += max(1,
self.chunk_size - self.chunk_overlap) #C
return chunksThe fixed-size implementation tokenizes the document, slices token windows, detokenizes each window, and advances by the chunk size minus overlap, with a minimum advance of one token. section 5.3.5
Choosing size and overlap
With a chunk size of 512 tokens and an overlap of 50, consecutive windows begin 462 tokens apart. The shared window can preserve a thought that crosses a boundary, at the cost of duplicated content. It does not make every boundary semantically coherent. section 5.3.5 section 5.2.2
Figure 5.4 Fixed-size chunking with and without overlap. A document is split into three chunks at rigid token boundaries, shown as stacked bars aligned to a token ruler. In the top panel (without overlap), hard cuts at exact token positions split a sentence about defective item returns across two chunks, leaving neither with the complete thought. In the bottom panel (with 50-token overlap), each consecutive pair of chunks shares a window of tokens, shown as the highlighted region where bars overlap. The boundary sentence now appears intact in Chunk 2, making it fully searchable. The overlap costs a small amount of duplicated storage but ensures that information at every boundary is preserved in at least one chunk.

The comparison illustrates how overlap can keep a boundary sentence intact in a retrievable chunk. section 5.3.5
Each chunk carries text, optional heading, offsets, and metadata. Structure-aware chunking can populate the heading; fixed-size chunking leaves it empty. The fixed-size implementation records token positions as offsets, so callers should not assume those values are character positions. section 5.3.5
Listing 5.9 Chunking strategy interface
class ChunkingStrategy(ABC):
def __init__(
self,
chunk_size: int = 512,
chunk_overlap: int = 50
): #A
self.chunk_size = chunk_size
self.chunk_overlap = chunk_overlap
@abstractmethod
def chunk(
self, document: ExtractedDocument
) -> List[Chunk]: #B
"""Break a document into chunks for embedding and retrieval."""
passA common strategy interface accepts an extracted document and returns chunks; size and overlap arrive through the constructor. This lets an index use a strategy suited to its content. section 5.3.5
Choose the strategy per index and apply it consistently during ingestion. Fixed-size blocks may merge self-contained FAQ entries, while long technical manuals benefit from boundaries that keep topics separate. Recursive splitting offers another approach: split at paragraph boundaries first, then split oversized pieces at finer boundaries. section 5.2.1 section 5.3.5









