A classifier assigns an observation to a predefined category. In text analysis, that might mean labeling a customer review positive, negative, or neutral. The business question comes first: which feedback needs attention, and how reliably can it be found? Extracting positive reviews with keywords is already a classification task, even without a more elaborate model. section 6.2 section 5.5.5
The categories depend on the decision:
- Customer-service emails can be sorted into complaints, inquiries, or requests. section 5.5.5
- Transactions can be classified as fraudulent or legitimate; user-generated content can be flagged as inappropriate. section 5.5.5
- A pair of texts can be classified by their relationship, using labels such as entailment, contradiction, or neutral. This is natural language inference, not sentiment analysis. section 6.2
Check a positive-review classifier
Suppose a keyword search marks reviews as positive. First remove rows without review comments. Then compare the keyword labels with reviews scored four or five stars: identify true positives, false positives, false negatives, and true negatives. Sensitivity asks how many score-positive reviews were found; specificity asks how many score-negative reviews were correctly left out. Inspect mismatches in the original records before changing the keywords or cleaning steps. A different generative-AI run may suggest different improvements, so measure the results rather than assuming the latest suggestion is better. section 5.5.5
Listing 5.7 Code snippet for assessing sensitivity and specificity
# Remove rows that don't have reviews.
df = df.dropna(subset = ['review_comment_message'])
# Extract records with positive reviews assessed by keywords and by review
scores.
posrev_keyword = df[df['positive_review']==True]
posrev_score = df[(df["review_score"]==5)|(df["review_score"]==4)]
# Perform set operations to determine true positives (TP), false positives
(FP), false negatives (FN) and true negatives (TN).
TP = pd.merge(posrev_keyword, posrev_score)
FP = posrev_keyword[posrev_keyword["review_id"].
isin(posrev_score["review_id"]) == False]
FN = posrev_score[posrev_score["review_id"].
isin(posrev_keyword["review_id"]) == False]
TN = df[(df["review_id"].isin(posrev_keyword["review_id"]) == False) &
(df["review_id"].isin(posrev_score["review_id"]) == False)]
# Calculate sensitivity and specificity
print("Sensitivity: ", round(len(TP) / (len(TP) + len(FN)),2))
print("Specificity: ", round(len(TN) / (len(TN) + len(FP)),2))
The Python listing shows the review-score comparison, including the record sets used to assess sensitivity and specificity. section 5.5.5
Classification quality matters beyond reviews: a missed complaint or an incorrectly flagged transaction can change what a team does next. Use the labels that fit the decision, and check both the cases the classifier catches and those it misses. section 5.5.5
