A text chunk is a smaller search unit cut from a document. Embedding a whole document can exceed a model’s token limit, blur its different topics into one vector, and return more text than an agent needs. Embedding chunks separately lets vector search retrieve the relevant portions. section 5.3.2
Choose boundaries and size with retrieval in mind:
- Too-small chunks lose context; too-large chunks mix topics. A starting range is 200–1,000 tokens, adjusted to the document. section 5.3.2
- Fixed-length cuts can split a thought. Share roughly 10%–20% of each chunk with its neighbor to preserve context across boundaries. section 5.3.2
Listing 5.3 Fixed-length chunking function
def fixed_length_chunking(text, chunk_size=500, overlap=50):
"""Split text into fixed-length chunks."""
chunks = []
start = 0
while start < len(text):
end = start + chunk_size
chunk = text[start:end].strip()
if chunk:
chunks.append(chunk)
start = end - overlap if end < len(text) else end #1
return chunks
The Python function slices from start to start + chunk_size, then advances to end − overlap until it reaches the document’s end. Keep overlap smaller than chunk_size so the start position advances. section 5.3.2
Retrieve relevant web-search passages
Fetch full web-search pages, split each page into 500-character chunks with 50 characters of overlap, and retain its title and URL with every chunk. Embed the chunk texts, then search for the three most similar to “quantum computing.” In the exercise, the selected passages used 304 tokens instead of the full results’ 37,312. section 5.3.5 section 5.6
