Tokenization breaks text into smaller units—usually words, punctuation marks, or phrases—so computers can process and analyze its structure. NLP systems use these tokens to examine word frequency, word types, co-occurrence, and text categories. section 2.4
Turning a sentence into tokens
The sentence “My name is Jeremy, and I like apples.” becomes separate tokens for each word and punctuation mark: My, Name, is, Jeremy, comma, and, I, like, apples, period. These tokens become building blocks for later text analysis. section 2.5
A practical word-frequency workflow combines tokenization with normalization:
- Read a text file, convert its contents to lowercase, and remove punctuation before counting words. section 2.5
- Tokenize the cleaned text, count unique and repeated words, and identify the most frequent terms. section 2.5
- Normalization prevents variants such as “truck,” “Truck,” and “truck!” from being counted as different words. section 2.5
Figure 2.11 Word-based tokenization breaks whole words and punctuation into chunks that can be processed and analyzed.
Shows word-based tokenization visually, including the separation of words and punctuation into processable chunks. section 2.4
