Fixed-length chunking divides text into pieces containing a predetermined number of characters, words, or tokens. It offers a straightforward way to prepare large corpora for retrieval-augmented generation when model context windows impose size limits. section 7.7.2 section 7.9
Use fixed-length chunking when implementation simplicity, speed, or scale outweighs precise linguistic boundaries:
- Choose a fixed number of words or tokens to produce predictable chunk sizes and straightforward processing. section 7.7.2
- Prefer it when context preservation is not critical, computational resources are limited, or the input is poorly structured. section 7.7.5
- Expect trade-offs: chunks can cut sentences in half, lose context, or require overlap to reduce missed boundaries. section 7.7.2 section 7.7.2
- Evaluate quality and performance against sentence, semantic, punctuation-based, or hierarchical chunking rather than assuming one strategy fits every corpus. section 7.7.2 section 7.7.5
A fixed-size view of the UK Constitution
Applying fixed-length chunking to the UK Constitution on Wikipedia produces uniform pieces, but some sentences are cut off and surrounding context disappears. An NLP-based strategy preserves more coherent boundaries, making the comparison useful when selecting a production approach. section 7.7.2
Table 7.3 Chunking approaches
|
Chunking approach
|
Description
|
|---|---|
|
Fixed length
|
Divide documents into chunks of a fixed number of words or tokens. This is straightforward but may sometimes split information that ideally should be kept together.
|
|
Sliding window
|
Use a fixed-sized sliding window with or without overlapping data. This can ensure that important boundaries within the text are not missed, but it can also lead to redundancy if there’s significant overlap.
|
|
Punctuation based
|
Divide the text based on punctuation, such as paragraphs or sections. This is less arbitrary than fixed-length chunking and often preserves the semantic integrity of the content. However, it can result in variable chunk sizes.
|
|
Topic or section breaks
|
In structured documents such as Wikipedia articles, natural breaks like sections or subsections can be used to define chunks. This method ensures that the content within a chunk is semantically coherent.
|
|
Adaptive
|
Use algorithms or models that adaptively determine the best way to chunk documents based on their content. This can be more complex but might yield semantically cohesive chunks.
|
Compare fixed length with sliding-window, punctuation-based, and section-based approaches, including their trade-offs around context, redundancy, semantic integrity, and chunk size. section 7.7.2
