A keyword search can miss a customer’s opinion when a review uses an inflected word but the search term or sentiment dictionary uses its base form. Lemmatization converts words to base forms using linguistic knowledge, making those forms easier to match. Unlike stemming, which reduces words by stripping inflections or suffixes, it relies on language analysis. section 5.5 section 5.6.5
This matters especially for a dictionary-based sentiment search: a dictionary may store nouns and verbs as lemmas, while reviews contain other forms. If the reviews are searched without lemmatization, that mismatch can contribute to missed positive reviews and low sensitivity. Applying the same kind of preprocessing to the reviews can bring them into closer alignment with the dictionary. section 5.6.5
A Portuguese-review workflow
Suppose a keyword search over Portuguese customer reviews misses expected matches. Ask an AI assistant whether preprocessing could help, then check its suggestions rather than applying every step. Lowercasing, removing punctuation, and trimming extra whitespace may be useful; removing stopwords may be irrelevant to a simple keyword search. To try lemmatization, load spaCy’s Portuguese model, pt_core_news_sm, process each review, and collect each token’s lemma_ into a cleaned review column. Search that column, then inspect sample matches and misses against the original reviews. section 5.5.5 section 5.6.5
Do not assume the change improves results merely because it is linguistically motivated. The demonstrated cleaning workflow uses stemming, while lemmatization is offered as an alternative and as a fix for a lemma-based dictionary. Test the chosen workflow on the dataset: compare search results, examine false positives and false negatives, and revisit the preprocessing if useful reviews still go unmatched. section 5.5.5 section 5.6.5
import spacy
nlp = spacy.load("pt_core_news_sm")
def lemmatize_text(text):
doc = nlp(text)
lemmatized_words = [token.lemma_ for token in doc]
return ' '.join(lemmatized_words)
def clean_text(text):
if not isinstance(text, str):
return ""
text = text.lower()
text = remove_punctuation(text)
text = remove_extra_whitespace(text)
text = lemmatize_text(text)
return text
df['cleaned_review_text'] = df['review_text'].apply(clean_text)
This alternative cleaning snippet shows where spaCy lemmatization fits before searching a cleaned review column; it is distinct from the stemming-based workflow. section 5.5.5
