Regular expressions match character patterns to split text into tokens, making them a practical first step before converting language into numeric inputs. section 14.2
From a sentence to tokens
Use "." to match each character, or combine "[\\w]+" with "[.,!?;]" to capture words and punctuation separately. Applied to "The quick brown fox jumped over the dog.", the latter produces tokens such as "The", "quick", "brown", and ".". section 14.2
Practical rules:
- Choose character tokens when individual symbols matter. section 14.2
- Choose word-and-punctuation patterns when preserving readable text units matters. section 14.2
- Index the resulting tokens separately; regex performs splitting, not numeric encoding. section 14.2
def split_words(text): return re.findall(r"[\w]+|[.,!?;]", text)
Inspect the compact pattern that separates word runs from punctuation. section 14.2
