NLTK is a Python library for practical text mining. It provides more consistent and specialized processing than generative AI in some tasks, especially when vocabulary, ambiguity, or repeatable rules matter. section 5.1.3
Use NLTK to prepare customer reviews before keyword detection:
- Remove punctuation, normalize case, and collapse extra whitespace. section 5.5.5
- Remove Portuguese stopwords when they add noise to the search. section 5.5.5
- Apply stemming with RSLP to improve matches across related word forms; compare it with lemmatization when appropriate. section 5.5.5
Cleaning Portuguese reviews
A review workflow can download NLTK's stopword or RSLP resources, clean the review_text column, and write the result to cleaned_review_text before applying the positive-keyword function. Inspect sample outputs and validate whether cleaning improves keyword results rather than assuming every suggested step helps. section 5.5.5
# Using NLTK for stemming
from nltk.stem import RSLPStemmer
nltk.download('rslp')
stemmer = RSLPStemmer()
def stem_words(text):
words = text.split()
stemmed_words = [stemmer.stem(word) for word in words]
return ' '.join(stemmed_words)
df['cleaned_review_text'] = df['cleaned_review_text'].apply(stem_words)
Shows the NLTK RSLPStemmer workflow for transforming tokenized text into stemmed words. section 5.5.5
