A tokenizer cuts text into countable pieces. Picture a coin sorter: tokens enter, matching tokens collect in bins, and the bin counts become a numerical vector. That vector can support search, keyword detection, and text comparison. Counting alone, however, discards word order. section 1.5.3 section 2.0
A token is not necessarily a whole word. Depending on the task, tokens can include punctuation, emojis, subword pieces, or individual characters. The tokenizer’s decisions determine which strings share a vocabulary entry—and which distinctions remain available downstream. section 2.1 section 8.1.2
Start with whitespace, then inspect the result
Python’s `"Trust me, though".split()` gives `['Trust', 'me,', 'though']`. It is a useful first pass, but `me,` will not match a keyword search for `me`. Inspecting the actual tokens is more informative than assuming that splitting has found the words you need. section 2.1.2
Give punctuation its own tokens
Try `re.findall(r"\w+(?:'\w+)?|[^\w\s]", "There's hope, maybe.")`. This rule retains `There's` while separating the comma and period. It handles common punctuation, not every contraction: the appropriate rule depends on the distinctions your application must preserve. section 2.1.3
Sometimes splitting a contraction is preferable. spaCy tokenizes `There's` as `There` and `'s`; its pipeline can also label tokens and detect sentence boundaries. For search or sentiment analysis, separating forms such as `is` and `n't` can make contraction expansion or synonym substitution possible. For authorship attribution, preserving exact spelling and style may matter more. section 2.1.4
>>> nlp.pipe_names #1
['tok2vec', 'tagger', 'parser', 'attribute_ruler', 'lemmatizer', 'ner']
>>> nlp = spacy.load('en_core_web_sm', disable=nlp.pipe_names)
>>> %timeit nlp(text)
199 ms ± 6.63 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Disabling unneeded spaCy pipeline components changes the measured run from about 4.67 seconds to 199 milliseconds on the example text. Benchmark the work you actually need: linguistic labels cost time, while a simpler regular-expression tokenizer can be faster when those labels are unnecessary. section 2.1.5
