A sentence boundary marks where one sentence ends and another begins. Finding it matters because later information-extraction steps depend on accurate sentence splits, yet punctuation alone is unreliable. section 11.2.1
A simple split on periods, exclamation points, or question marks fails around nested quotations, abbreviations, ellipses, technical notation, informal messages, and punctuation without surrounding spaces. Practical segmentation therefore uses either manually programmed patterns or statistical NLP models. section 11.2.1
A regular-expression segmenter can improve on naive splitting, but each rule trades coverage against accuracy:
- Match sentence-ending punctuation followed by whitespace or the end of the string; this handles repeated punctuation but can consume the terminator. section 11.2.2
- Protect periods inside nested quotes, but recognize that whitespace-based rules may incorrectly join sentences whose final punctuation is quoted. section 11.2.2
- Use lookarounds to split periods without spaces while avoiding periods inside numbers, as in compact text such as “GT.You?”. section 11.2.2
- Compare regex output with a statistical or pipeline-based segmenter on representative text rather than assuming either method is universally correct. section 11.2.2
Regex versus a sentence recognizer
A spaCy pipeline segments “Are you an M.D. Dr. Gebru? either way you are brilliant.” into two sentences despite the abbreviation. Enabling only its sentence recognizer reduces processing time compared with a fuller pipeline, while a comparison on body text reports 76% agreement with regex segmentation.
