Tokenization divides text into model-ready units rather than simply splitting on spaces. A tokenizer may separate “don’t” into “do” and “n’t”, or break uncommon words into meaningful subwords. section 1.3.2
Tokenization is model-specific: applying one model’s tokenizer to another model’s input can produce misleading fragments. Long inputs also risk losing earlier tokens when they exceed the context window. section 1.3.2
A practical text-analysis workflow typically combines tokenization with:
- Lowercasing text and removing punctuation-like tokens. section 5.3.4
- Removing language-appropriate stopwords before counting terms. section 5.3.4
- Using stemming or lemmatization when keyword matching needs broader linguistic forms. section 5.5
Portuguese customer reviews
For a DataFrame column named review_comment_message, tokenize reviews with a Portuguese-aware tokenizer, filter Portuguese stopwords, and count terms with Python. An initial generated workflow may silently use English defaults, so inspect and correct the tokenizer configuration before trusting the frequencies. section 5.3.4
import pandas as pd
import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from collections import Counter
# Make sure to download Portuguese stopwords
nltk.download('stopwords')
nltk.download('punkt')
# Load your DataFrame
df = pd.read_csv('olist_order_reviews_dataset.csv')
# Preprocessing function
def preprocess_text(text):
# Tokenize, convert to lowercase, and remove punctuation
tokens = word_tokenize(text.lower(), language='portuguese')
words = [word for word in tokens if word.isalpha()]
# Remove Portuguese stopwords
stop_words = set(stopwords.words('portuguese'))
filtered_words = [word for word in words if word not in stop_words]
return filtered_words
# Apply preprocessing to the 'review_comment_message' column
df['processed_review'] =
df['review_comment_message'].apply(preprocess_text)
# Calculate word frequencies
word_freq = Counter()
for _, row in df.iterrows():
word_freq.update(row['processed_review'])
# Print the most common words and their frequencies
print(word_freq.most_common(10))
Browse the corrected Python workflow for Portuguese tokenization, stopword removal, and review-word frequency counting. section 5.3.4
