A search or chatbot pipeline may need to treat related word forms alike. Lemmatization maps a word to a lemma: a base form chosen using information about words and their meanings, rather than just trimming endings. Processing the resulting tokens can reduce the vocabulary a system must handle. section 2.3.2
A stemmer applies spelling rules and may produce a form that is not a dictionary word. A lemmatizer can use grammatical information: with WordNet, “better” stays “better” by default but becomes “good” when marked as an adjective. The choice of tool and word context therefore matter. section 2.3.2
>>> nltk.download('wordnet')
True
>>> nltk.download('omw-1.4')
True
>>> from nltk.stem import WordNetLemmatizer
>>> lemmatizer = WordNetLemmatizer()
>>> lemmatizer.lemmatize("better") #1
'better'
>>> lemmatizer.lemmatize("better", pos="a") #2
'good'
>>> lemmatizer.lemmatize("good", pos="a")
'good'
>>> lemmatizer.lemmatize("goods", pos="a")
'goods'
>>> lemmatizer.lemmatize("goods", pos="n")
'good'
>>> lemmatizer.lemmatize("goodness", pos="n")
'goodness'
>>> lemmatizer.lemmatize("best", pos="a")
'best'
Run the WordNet example to inspect how part-of-speech settings change individual results. “Goods” becomes “good” when marked as a noun, while “goodness” remains unchanged. section 2.3.2
Use the grouping deliberately:
- For search, consolidating forms can help retrieve documents despite differences in wording, but normalization can also reduce precision. section 2.7 section 2.8
- For a chatbot, fewer distinct forms can simplify response rules, yet treating related forms alike may erase distinctions among “chat,” “chatter,” and “chatty.” section 2.3.2
- For TF–IDF similarity, lemmatization does not solve meaning: it may miss synonyms such as “beautiful” and “pretty” or group words with opposing meanings. section 4.1.1
Do not assume an English-style word-ending procedure transfers to every writing system. Chinese character components can change meaning when separated, although the plural suffix 们 offers a different kind of removable form. Inspect actual tokens and downstream results before adopting a normalization rule. section 2.4.1 section 2.3.2
