Vocabulary normalization combines tokens that mean or function similarly under a shared form. A smaller vocabulary needs fewer retained tokens and can strengthen associations across different spellings, but every merge discards a distinction. section 2.3.2
Choose a transformation that fits the distinctions your pipeline needs:
- Case folding maps “House” to “house.” It reduces capitalization variants, but may erase the difference between “doctor” and the proper noun “Doctor.” section 2.3.2
- Stemming strips word endings. Porter stemming turns “fairly” into “fairli,” while Snowball stemming produces “fair.” section 2.3.2
- Lemmatization can depend on part of speech: WordNet maps “better” to “good” when given the adjective tag, but leaves it unchanged without that tag. section 2.3.2
>>> nltk.download('wordnet')
True
>>> nltk.download('omw-1.4')
True
>>> from nltk.stem import WordNetLemmatizer
>>> lemmatizer = WordNetLemmatizer()
>>> lemmatizer.lemmatize("better") #1
'better'
>>> lemmatizer.lemmatize("better", pos="a") #2
'good'
>>> lemmatizer.lemmatize("good", pos="a")
'good'
>>> lemmatizer.lemmatize("goods", pos="a")
'goods'
>>> lemmatizer.lemmatize("goods", pos="n")
'good'
>>> lemmatizer.lemmatize("goodness", pos="n")
'goodness'
>>> lemmatizer.lemmatize("best", pos="a")
'best'
Try the WordNet listing to see how supplying a part-of-speech tag changes which forms are combined. section 2.3.2
