Sentence splitting, or sentence segmentation, divides text into individual sentence units and identifies their boundaries. It isolates coherent packets of meaning, making them useful for extracting facts, selecting dialogue, and training chatbots on longer texts. section 11.2
A practical segmentation pipeline must account for punctuation that does not mark sentence endings:
- Avoid relying on split('.!?'): periods, question marks, and exclamation points can appear inside quotations, nested dialogue, abbreviations, and technical notation. section 11.2.1
- Approach the problem with manually programmed regular expressions and pattern matching, or with statistical models trained from data. section 11.2.1
- Inspect and correct results when needed: even extracted sentences from technical text can contain boundary errors. section 11.2.1
Quoted dialogue and nested punctuation
In “She yelled ‘It’s right here!’ but I kept looking,” the exclamation mark belongs to quoted speech rather than necessarily ending the surrounding sentence. Similar ambiguity appears in questions inside quotations and nested stories. section 11.2.1
