Hierarchical chunking divides text according to nested structural levels, such as chapters, sections, and subsections. It treats document organization as a guide for creating retrieval units. section 7.7.2
Use structural boundaries alongside other chunking choices:
- Compare nested section boundaries with sentence, fixed-length, token-based, and semantic splitting. section 7.7.2
- Account for trade-offs: fixed-length splitting can cut information and lose context, while structural breaks preserve document organization but may produce variable chunk sizes. section 7.7.2
Table 7.3 Chunking approaches
|
Chunking approach
|
Description
|
|---|---|
|
Fixed length
|
Divide documents into chunks of a fixed number of words or tokens. This is straightforward but may sometimes split information that ideally should be kept together.
|
|
Sliding window
|
Use a fixed-sized sliding window with or without overlapping data. This can ensure that important boundaries within the text are not missed, but it can also lead to redundancy if there’s significant overlap.
|
|
Punctuation based
|
Divide the text based on punctuation, such as paragraphs or sections. This is less arbitrary than fixed-length chunking and often preserves the semantic integrity of the content. However, it can result in variable chunk sizes.
|
|
Topic or section breaks
|
In structured documents such as Wikipedia articles, natural breaks like sections or subsections can be used to define chunks. This method ensures that the content within a chunk is semantically coherent.
|
|
Adaptive
|
Use algorithms or models that adaptively determine the best way to chunk documents based on their content. This can be more complex but might yield semantically cohesive chunks.
|
Compare hierarchical or topic-based boundaries with fixed-length, sliding-window, and punctuation-based approaches when selecting a retrieval strategy. section 7.7.2
