Stemming is a heuristic, rule-based text-cleaning technique that reduces inflected or derived words to a simplified stem. It helps map related forms toward a shared representation before machine-learning processing, but the result may not be a valid word. section 7.6 section 7.20
A practical stemming workflow follows the broader text-preparation pipeline:
- Clean unwanted characters and punctuation, normalize case, and tokenize the text before stemming. section 7.12
- Pass each token to an NLTK stemmer such as PorterStemmer, then inspect the resulting stems. section 7.12
- Expect approximate spellings because suffix-removal rules can produce stems that are not dictionary words. section 7.6
- Choose lemmatization instead when dictionary or canonical forms and better linguistic accuracy matter more than processing time. section 7.6
Tokenize, then stem
Using NLTK's PorterStemmer on “eats eating studies study” produces stems for each token; “studies” becomes “studi,” while “studying” becomes “study.” Lemmatization would produce the dictionary form “study” for “studies.” section 7.12 section 7.6
import nltk
from nltk.stem import PorterStemmer
stem = PorterStemmer()
text = "eats eating studies study"
tokenization = nltk.word_tokenize(text)
for word in tokenization:
print("Stem for {} is {}".format(word, stem.stem(word)))
Run the NLTK example to inspect token-by-token Porter stemming output. section 7.12
