Keyword extraction identifies the main words or topics in short texts when a full summary would add little value. Customer reviews are a practical fit, especially alongside sentiment analysis. section 6.3.4
Common approaches include:
- Frequency-based extraction, which ranks recurring terms after tokenization and stopword removal. section 6.3.4
- Rule-based scoring, including sentence position, keyword presence, and TF-IDF. section 6.3.4
- TextRank, which extracts central content from overlap relationships between text units. section 6.3.4
Review keywords
A review praising an online store produced “congratulations,” “Lannister,” and “stores” through frequency-based extraction, while a review about a mislabeled device produced “efficient,” “brand,” and “device.” These outputs require inspection because frequent words are not always the most useful topics. section 6.3.4
Listing 6.6 Summarizing the longest review—a frequency-based approach
import pandas as pd
import nltk
import string
from collections import Counter
# Download the required NLTK resources
nltk.download('punkt')
nltk.download('stopwords')
# Sample data (manually adapted to remove empty records)
df = pd.read_csv('olist_order_reviews_dataset.csv')
df = df.dropna(subset = ['review_comment_message'])
# Function to tokenize and remove stopwords
def preprocess(text):
stopwords = nltk.corpus.stopwords.words('portuguese')
tokens = nltk.word_tokenize(text.lower())
tokens = [token for token in tokens if token not in string.punctuation
and token not in stopwords]
return tokens
# Function to create word frequency distribution
def word_frequency(tokens):
frequency = Counter(tokens)
return frequency
# Function to summarize short reviews
def summarize_reviews(text, num_keywords=3):
tokens = preprocess(text)
frequency = word_frequency(tokens)
important_words = [word for word, count in
frequency.most_common(num_keywords)]
summary = ' '.join(important_words)
return summary
# Apply the function to the DataFrame
df['summary'] = df['review_comment_message'].apply(summarize_reviews)
# Display the results (manually adapted to print the summary of the longest
message)
print("Longest review:", df.loc[1316]["review_comment_message"])
print("Summary:", df.loc[1316]["summary"])
The workflow loads review data, removes empty records, tokenizes Portuguese text, removes punctuation and stopwords, counts terms, and selects keywords. section 6.3.4
