A tokenizer segments text into discrete tokens using processing rules. In frequency analysis, tokenization determines which words are counted, so the rules should match the language of the reviews. section 5.3.4
Portuguese customer reviews
For a DataFrame column named review_comment_message, a practical workflow lowercases reviews, tokenizes them, keeps alphabetic tokens, removes Portuguese stopwords, and counts the remaining words. A generic tokenizer may silently use English defaults even after Portuguese data is specified; inspect the generated code rather than assuming it adapted correctly. section 5.3.4
import pandas as pd
import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from collections import Counter
# Make sure to download Portuguese stopwords
nltk.download('stopwords')
nltk.download('punkt')
# Load your DataFrame
df = pd.read_csv('olist_order_reviews_dataset.csv')
# Preprocessing function
def preprocess_text(text):
# Tokenize, convert to lowercase, and remove punctuation
tokens = word_tokenize(text.lower(), language='portuguese')
words = [word for word in tokens if word.isalpha()]
# Remove Portuguese stopwords
stop_words = set(stopwords.words('portuguese'))
filtered_words = [word for word in words if word not in stop_words]
return filtered_words
# Apply preprocessing to the 'review_comment_message' column
df['processed_review'] =
df['review_comment_message'].apply(preprocess_text)
# Calculate word frequencies
word_freq = Counter()
for _, row in df.iterrows():
word_freq.update(row['processed_review'])
# Print the most common words and their frequencies
print(word_freq.most_common(10))
Compare the language-aware NLTK call with a default tokenizer, then run the workflow against the actual review column and validate the resulting counts. section 5.3.4
