Frequency measures how often a word or expression appears in a language or a text collection. In text analysis, counts become more useful when they preserve context: “delivery” says little alone, while frequent phrases such as “failed delivery” or “super fast delivery” can reveal customer concerns or strengths. section 5.3.5 section 5.6.1
Use frequency analysis to:
- Count common words when a broad view of the text is sufficient. section 5.3.5
- Count consecutive word sequences, called n-grams, when context and nuance matter. section 5.3.5
- Compare bigrams, trigrams, or higher-order n-grams according to the desired level of detail. section 5.3.5
Review analysis workflow
A practical workflow loads review data, tokenizes Portuguese text, removes punctuation and Portuguese stopwords, and counts trigrams. Inspect the resulting phrases before drawing conclusions, since recurrence can indicate trends or concerns but still requires interpretation. section 5.3.5
Listing 5.1 Code snippet for analyzing trigrams
import pandas as pd
import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from nltk.collocations import TrigramAssocMeasures,
TrigramCollocationFinder
from collections import Counter
# Make sure to download Portuguese stopwords
nltk.download('stopwords')
nltk.download('punkt')
# Load your DataFrame
df = pd.read_csv('olist_order_reviews_dataset.csv')
# Preprocessing function
def preprocess_text(text):
# Tokenize, convert to lowercase, and remove punctuation
tokens = word_tokenize(text.lower(), language='portuguese')
words = [word for word in tokens if word.isalpha()]
# Remove Portuguese stopwords
stop_words = set(stopwords.words('portuguese'))
filtered_words = [word for word in words if word not in stop_words]
return filtered_words
# Apply preprocessing to the 'review_comment_message' column
df['processed_review'] =
df['review_comment_message'].apply(preprocess_text)
# Calculate trigram frequencies
trigram_measures = TrigramAssocMeasures()
trigram_freq = Counter()
for _, row in df.iterrows():
finder = TrigramCollocationFinder.from_words(row['processed_review'])
trigram_freq.update(finder.ngram_fd)
# Print the most common trigrams and their frequencies
print(trigram_freq.most_common(10))
Inspect the Python listing for a concrete trigram-analysis implementation using pandas and NLTK. section 5.3.5
