Chunk granularity is the amount of text placed in each segment for embedding and retrieval. Larger chunks carry more context but can add noise; smaller chunks can omit details needed to answer a query. section 3.6.7 section 4.5.4
Tune chunk size alongside retrieval settings:
- Compare sizes such as 100, 300, 500, 700, and 900 characters, and vary overlap and the number of retrieved chunks. section 3.6.7
- Evaluate configurations against varied test data; a size that supplies enough context without overload must be established experimentally. section 3.6.7
- For policy documents, 500-character chunks with 100-character overlap balance specificity with context across chunk boundaries. section 4.5.4
Testing chunk sizes
Run separate RAG experiments for 100, 300, 500, 700, and 900 characters, then compare their performance rather than assuming one size will work best. section 3.6.7
chunk_sizes = [100, 300, 500, 700, 900]
for chunk_size in chunk_sizes:
experiment_name = f"chunk-size-{chunk_size}"
run_experiment(...)
The experiment loop names each run by chunk size, making configuration comparisons straightforward. section 3.6.7
