Document chunking divides text into smaller segments before embedding and retrieval. Narrower chunks produce more specific embeddings, helping a retriever return information that better matches a query. section 2.1.1
Chunk size depends on the data, domain, and use case. Smaller chunks can improve retrieval and entity detection, but increasing the number of indexed segments also requires experimentation and evaluation. section 2.1.1 section 7.2.1
Choose chunk boundaries and overlap deliberately:
- Split at sentences, paragraphs, semantic boundaries, fixed character counts, or whitespace according to the retrieval problem. section 2.1.1
- Use overlap when an answer may span adjacent chunks; a 500-character window with 40 characters of overlap improves continuity at the cost of a larger index. section 2.2.3
- Prefer whitespace boundaries when splitting text so chunks do not begin or end with broken words. section 2.2.3
- Measure token sizes and test alternatives rather than assuming one chunk size works everywhere. section 2.2.3 section 7.2.1
Paper and book chunking
A paper can be split into several-hundred-character chunks with a 500-character window and 40-character overlap. A book corpus can be cleaned, divided into books, and chunked with a 1,000-word limit and 40-word overlap. section 2.2.3 section 7.2.1
Listing 2.1 The text-chunking function
def chunk_text(text, chunk_size, overlap, split_on_whitespace_only=True): #1
chunks = []
index = 0
while index < len(text):
if split_on_whitespace_only:
prev_whitespace = 0
left_index = index - overlap
while left_index >= 0:
if text[left_index] == " ":
prev_whitespace = left_index
break
left_index -= 1
next_whitespace = text.find(" ", index + chunk_size)
if next_whitespace == -1:
next_whitespace = len(text)
chunk = text[prev_whitespace:next_whitespace].strip()
chunks.append(chunk)
index = next_whitespace + 1
else:
start = max(0, index - overlap + 1)
end = min(index + chunk_size + overlap, len(text))
chunk = text[start:end].strip()
chunks.append(chunk)
index += chunk_size
return chunks
chunks = chunk_text(text, 500, 40) #2
print(len(chunks)) # 89 chunks in total #3
Inspect a Python implementation that supports configurable chunk size, overlap, and whitespace-only splitting. section 2.2.3
