A regular expression tokenizer divides text into tokens by applying pattern rules. It favors speed over the linguistic accuracy and additional tags provided by fuller NLP pipelines. section 2.1.5
A compact Python tokenizer
The pattern r'\w+(?:\'\w+)?|[^\w\s]' keeps word-internal apostrophes while separating punctuation: re.findall(pattern, text) produced about 20,000 tokens and ran in 8.77 milliseconds on the demonstrated text. section 2.1.5
Use the technique when:
- Processing large collections where tokenizer throughput matters more than linguistic annotation. section 2.1.5
- Replacing a slow full spaCy pipeline when embeddings, tagging, parsing, lemmatization, or named entities are unnecessary. section 2.1.5
- Accepting reduced accuracy in exchange for substantial processing-time reductions, including medical-record parsing at scale. section 11.2.2
>>> pattern = r'\w+(?:\'\w+)?|[^\w\s]'
>>> tokens = re.findall(pattern, text) #1
>>> f'{round(len(tokens) / 10_000)}0k'
'20k'
>>> %timeit re.findall(pattern, text)
8.77 ms ± 29.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Inspect the runnable regular-expression tokenization benchmark and its resulting token count. section 2.1.5
>>> re.split(r'[!.?]+[\s$]+', ... "Hello World.... Are you there?!?! I'm going to Mars!") ['Hello World', 'Are you there', "I'm going to Mars!"]
Inspect a related regular-expression sentence split and see how punctuation rules affect boundaries. section 11.2.2
