Suppose you want to find the most common words in customer reviews. Words such as “and,” “the,” and “a” may appear often without telling you much about the products. These common, low-information words are stop words. Removing them during text cleaning can make a word-frequency list more useful. section 5.2.1 section 5.3.4
The language matters. For Portuguese reviews, use Portuguese stop words and a Portuguese tokenizer. First split each review into words, convert them to lowercase, and filter out punctuation and stop words; then count the words that remain. If you ask an AI assistant for Python code, specify the DataFrame column and the review language. Inspect the result: a generated example selected Portuguese stop words but left its tokenizer on English defaults. section 5.3.4
import pandas as pd
import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from collections import Counter
# Make sure to download Portuguese stopwords
nltk.download('stopwords')
nltk.download('punkt')
# Load your DataFrame
df = pd.read_csv('olist_order_reviews_dataset.csv')
# Preprocessing function
def preprocess_text(text):
# Tokenize, convert to lowercase, and remove punctuation
tokens = word_tokenize(text.lower(), language='portuguese')
words = [word for word in tokens if word.isalpha()]
# Remove Portuguese stopwords
stop_words = set(stopwords.words('portuguese'))
filtered_words = [word for word in words if word not in stop_words]
return filtered_words
# Apply preprocessing to the 'review_comment_message' column
df['processed_review'] =
df['review_comment_message'].apply(preprocess_text)
# Calculate word frequencies
word_freq = Counter()
for _, row in df.iterrows():
word_freq.update(row['processed_review'])
# Print the most common words and their frequencies
print(word_freq.most_common(10))
The corrected Python example explicitly sets the tokenizer language to Portuguese and filters tokens using NLTK’s Portuguese stop-word list. section 5.3.4
Do not remove stop words just because a cleaning checklist suggests it. For a simple keyword search, the extra step may be irrelevant. Decide what you are trying to detect, inspect the cleaned reviews, and keep only the preprocessing steps that help that task. section 5.5.5
