A search for “houses” might also need to find “house.” Stemming groups related word forms under a shared token by removing endings such as plural, possessive, or verb suffixes. The resulting stem need not be a correctly spelled word; it is a label for the grouped forms. section 2.3.2
Apply the same normalization to indexed documents and incoming queries. Grouping forms reduces vocabulary size and can increase search recall, but it may also return results the user did not want. Whether that trade-off helps depends on which distinctions the application needs to preserve. section 2.3.2 section 2.8
>>> def stem(phrase):
... return ' '.join([re.findall('^(.*ss|.*?)(s)?$',
... word)[0][0].strip("'") for word in phrase.lower().split()])
>>> stem('houses')
'house'
>>> stem("Doctor House's calls")
'doctor house call'
This small Python function lowercases a phrase and uses a suffix rule to turn “houses” into “house.” It illustrates the mechanism, not a complete stemming strategy. section 2.3.2
Compare two stemmers
NLTK’s Porter stemmer maps “fairly” to “fairli,” while its Snowball stemmer maps it to “fair.” Both reduce “washed” to “wash” and “dishes” to “dish.” The comparison makes clear why a stem is a matching token, not necessarily a word to display to users. section 2.3.2
>>> from nltk.stem.porter import PorterStemmer
>>> stemmer = PorterStemmer()
>>> ' '.join([stemmer.stem(w).strip("'") for w in
... "dish washer's fairly washed dishes".split()])
'dish washer fairli wash dish'
Run this NLTK PorterStemmer example to inspect the stem of each word in a short phrase. section 2.3.2
Stemming is not interchangeable with lemmatization: a lemmatizer can use word information such as part of speech, while a stemmer can produce a nonword. Neither should be applied blindly. Word-building patterns differ across languages; splitting a Chinese character into radicals, for example, can radically change its meaning. section 2.3.2 section 2.4.1
