Lemmatization reduces a word to its dictionary or canonical form, called a lemma. In text cleaning, it helps bring different forms of a word toward a common form before the text is represented for analysis. section 7.6
Unlike stemming, which removes endings by rule and can produce forms such as “studi,” lemmatization aims for a dictionary word such as “study.” It generally gives better results but takes more time. section 7.6 section 7.12
A practical preprocessing sequence is:
- Clean unwanted punctuation, normalize case, and tokenize the text into words. section 7.12
- Remove stop words where appropriate, then lemmatize the remaining words. section 7.12 section 7.15
import nltk
from nltk.stem import WordNetLemmatizer
wordnet_lemmatizer = WordNetLemmatizer()
text = "eats eating studies study"
tokenization = nltk.word_tokenize(text)
for word in tokenization:
print("Lemma for {} is {}".format(word, wordnet_lemmatizer.lemmatize(word)))
Tokenize a short string with NLTK, pass each token to WordNetLemmatizer, and inspect the printed word–lemma pairs. Compare them with the stemming example to see why the resulting spelling matters. section 7.12
from textblob import Word reviewData.body = reviewData.body.apply(lambda x: " ".join([Word(word).lemmatize() for word in x.split()])) reviewData.body.head()
For customer reviews, apply TextBlob’s Word.lemmatize() to words in each cleaned review body and inspect the resulting DataFrame column before calculating sentiment. section 7.15
