Lemmatization
Lemmatization is the linguistic normalization of inflected word forms to their canonical dictionary forms, called lemmas.
Natural Language Processing in Action, Second Edition
A lemmatizer is a text-processing component that maps inflected word forms to their dictionary base forms.
- How it is used
- Groups related word forms for search, chatbot responses, and token-count analysis.
- How deep it goes
- Defines lemmas and demonstrates library outputs and part-of-speech settings, but does not implement a lemmatizer from scratch.
- Examples
- Compares WordNet outputs for “better” and “goods” under different part-of-speech settings and shows spaCy lemma outputs.
- What you can do after
- Apply it via NLTK WordNetLemmatizer or spaCy, then evaluate whether grouped forms improve retrieval without hiding important distinctions.
Read how Natural Language Processing in Action, Second Edition explains lemmatization →
Starting Data Analytics with Generative AI and Python
Lemmatization is the process of converting inflected words into their canonical dictionary forms using linguistic analysis.
- How it is used
- Aligns customer-review words with search terms or lemma-based sentiment dictionaries to reduce missed matches.
- How deep it goes
- Defines the technique and sketches spaCy preprocessing, but does not detail its linguistic internals.
- Examples
- Uses Portuguese customer reviews, a lemma-based sentiment dictionary, and a spaCy Portuguese-model preprocessing snippet.
- What you can do after
- Apply it via spaCy to preprocess reviews for keyword or dictionary searches, then check whether missed matches decrease.
Read how Starting Data Analytics with Generative AI and Python explains lemmatization →
Data Without Labels
Lemmatization is a linguistic normalization method that reduces a word to its dictionary or canonical form.
- How it is used
- Normalizes word forms during text cleaning before representation or sentiment analysis.
- How deep it goes
- Defines the lemma, compares results and processing time with stemming, and demonstrates Python application without explaining linguistic internals.
- Examples
- Uses word-form comparisons, an NLTK token-and-lemma example, and lemmatized customer reviews.
- What you can do after
- Apply it via NLTK WordNetLemmatizer or TextBlob Word.lemmatize() in a Python text-cleaning workflow.
Related concepts
- stemmingalternative to · Stemming is a text-normalization technique that heuristically reduces inflected or derived word forms to a shared stem, which may not be a valid word.
- synonymcontrasts with · A synonym is a word or expression with the same or a closely related meaning as another word or expression, particularly in a given context.
- lexicon normalizationpart of · Lexicon normalization maps variant word forms to a common root or canonical form while preserving their shared lexical meaning.
- lexical searchuses · Lexical search retrieves or ranks items by matching query terms or other surface-text features against indexed content, often using sparse representations and statistical weighting.
- precisionproduces · Precision is the proportion of items identified as positive, selected, or retrieved that satisfy the relevant correctness or relevance criterion.
- spaCyuses · spaCy is an open-source Python library for natural language processing, providing tools and pretrained pipelines for tasks such as tokenization and linguistic analysis.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.