A sentence splitter divides text at sentence boundaries so retrieval-augmented generation chunks retain complete sentences and more coherent meaning. Common boundaries include periods, question marks, and exclamation points. section 7.7.4
Choose the implementation according to the required accuracy and chunk-size control:
- Use a regular-expression split such as '[.!?]' for a fast, simple sentence list; strip whitespace and discard empty results. section 7.7.4
- Use punctuation-based splitting when preserving semantic integrity matters more than producing uniform chunk sizes. section 7.7.2
- Treat textwrap.wrap() cautiously: it controls width and avoids breaking words when configured, but it was designed for display formatting rather than sentence detection. section 7.7.4
Minimal implementation
A practical starting point is: re.split('[.!?]', text), followed by stripping whitespace and filtering empty strings. For production retrieval, test the result against long sentences and domain-specific punctuation before relying on it. section 7.7.4
Listing 7.1 Split sentence function
def split_sentences(text):
sentences = re.split('[.!?]', text) #1
sentences = [sentence.strip() for sentence in sentences if sentence]
return sentences
Inspect the complete split_sentences function and its cleanup steps. section 7.7.4
