Tokenization breaks text into discrete units called tokens. It is a basic step in preparing text for analysis: tokens can be collected into a vocabulary—the set of unique tokens in a corpus—and used in text representations such as bag of words. section 7.8 section 7.7
Choose token boundaries to suit the task:
- Word tokenization separates words; punctuation may appear as separate tokens unless it is removed first. section 7.8 section 7.12
- Subword tokenization can split “raining” into “rain” and “ing”; character tokenization splits it into individual letters. section 7.8
- Available approaches also split on whitespace or blank lines, or use patterns to identify tokens and separators. section 7.8
import nltk text_d = "Hey!!! How are you doing? And how is your health! Bye, take care." nltk.tokenize.word_tokenize(text_d)
Run the NLTK example to inspect how word_tokenize separates a short sentence, including its punctuation marks. section 7.12
Inspect and clean the input according to the intended representation. The Python examples normalize spaces, remove punctuation or lowercase text before tokenizing when those changes are wanted. Once words are tokenized, they can be filtered for stop words or processed with stemming or lemmatization. Removing punctuation is a choice, not a requirement: the uncleaned example retains punctuation as tokens. section 7.12
Customer-review sentiment
A review workflow lowercases and cleans review text, removes stop words and frequent words, tokenizes a review with NLTK, and lemmatizes the text before calculating sentiment with TextBlob. Tokenization is one preparation step; the sentiment score comes later. section 7.15
