NLP-based chunking divides text along linguistic boundaries, such as sentences or phrases, helping RAG systems preserve context. NLTK can split sentences; spaCy also supports grouping words into meaningful units. section 7.7.5
Choose an approach with the task and operating constraints in mind:
- Prefer linguistic boundaries when context and language nuances matter or documents are well structured. section 7.7.5
- Consider fixed-length chunking when performance, resources, or unclear boundaries make linguistic processing less practical. spaCy is described as efficient for large-scale processing; NLTK may be slower. section 7.7.5
Listing 7.5 Sentence chunking using spaCy
def split_sentences_by_spacy(text, max_tokens, overlap=0, model="en_core_web_sm"):
nlp = spacy.load(model) #1
doc = nlp(text) #2
sentences = [sent.text for sent in doc.sents]
tokens_lengths = [count_tokens(sent) for sent in sentences] #3
chunks = []
start_idx = 0
while start_idx < len(sentences):
current_chunk = []
current_token_count = 0
for idx in range(start_idx, len(sentences)):
if current_token_count + tokens_lengths[idx] > max_tokens:
break
current_chunk.append(sentences[idx])
current_token_count += tokens_lengths[idx]
chunks.append(" ".join(current_chunk))
if overlap >= len(current_chunk): #4
start_idx += 1
else:
start_idx += len(current_chunk) - overlap
return chunks
See a spaCy example that groups sentences up to a token limit and supports overlap between chunks. section 7.7.5
