Token-based splitting divides text into chunks containing a fixed number of tokens, such as words. It offers finer control than sentence splitting, but may cut sentences and remove context. section 7.7.2
Apply it as one option in a retrieval-augmented generation chunking strategy:
- Choose a token limit that fits the corpus, query patterns, and application requirements. section 7.7.2
- Test the resulting retrieval behavior because fixed token boundaries can split information or omit context. section 7.7.2
Constitutional text
A UK Constitution example contrasts boundary-based outcomes: fixed-size chunks can cut information and lose context, while NLP-based segmentation better preserves meaning. Token-based splitting has the same boundary risk when a sentence crosses the token limit. section 7.7.2
Table 7.3 Chunking approaches
|
Chunking approach
|
Description
|
|---|---|
|
Fixed length
|
Divide documents into chunks of a fixed number of words or tokens. This is straightforward but may sometimes split information that ideally should be kept together.
|
|
Sliding window
|
Use a fixed-sized sliding window with or without overlapping data. This can ensure that important boundaries within the text are not missed, but it can also lead to redundancy if there’s significant overlap.
|
|
Punctuation based
|
Divide the text based on punctuation, such as paragraphs or sections. This is less arbitrary than fixed-length chunking and often preserves the semantic integrity of the content. However, it can result in variable chunk sizes.
|
|
Topic or section breaks
|
In structured documents such as Wikipedia articles, natural breaks like sections or subsections can be used to define chunks. This method ensures that the content within a chunk is semantically coherent.
|
|
Adaptive
|
Use algorithms or models that adaptively determine the best way to chunk documents based on their content. This can be more complex but might yield semantically cohesive chunks.
|
Compare token-based splitting with sliding-window, punctuation-based, and topic-or-section approaches by boundary control, overlap, semantic integrity, and chunk-size variability. section 7.7.2
