spaCy is a Python NLP library used for context-aware text segmentation in retrieval-augmented generation. It can split documents at linguistic boundaries such as sentences, clauses, phrases, or entities. section 7.7.5
A spaCy-based chunking pipeline can:
- Load a language model such as en_core_web_sm and process text into a document. section 7.7.5
- Extract sentence boundaries from doc.sents and count tokens for each sentence. section 7.7.5
- Assemble chunks under a maximum token limit while preserving configurable sentence overlap. section 7.7.5
Listing 7.5 Sentence chunking using spaCy
def split_sentences_by_spacy(text, max_tokens, overlap=0, model="en_core_web_sm"):
nlp = spacy.load(model) #1
doc = nlp(text) #2
sentences = [sent.text for sent in doc.sents]
tokens_lengths = [count_tokens(sent) for sent in sentences] #3
chunks = []
start_idx = 0
while start_idx < len(sentences):
current_chunk = []
current_token_count = 0
for idx in range(start_idx, len(sentences)):
if current_token_count + tokens_lengths[idx] > max_tokens:
break
current_chunk.append(sentences[idx])
current_token_count += tokens_lengths[idx]
chunks.append(" ".join(current_chunk))
if overlap >= len(current_chunk): #4
start_idx += 1
else:
start_idx += len(current_chunk) - overlap
return chunks
The listing demonstrates model loading, sentence extraction, token-aware chunk construction, and overlap management. section 7.7.5
Choosing spaCy for chunking
Use spaCy when retrieval or generation depends on preserving linguistic context. Compared with fixed-length chunking, it may require more computational resources and domain knowledge; the comparison records four spaCy chunks, 5.8 seconds of execution time, and 5,876 tokens. section 7.7.5
