When a classifier finds positive customer reviews, the practical question is: how many of the positive reviews did it catch? Recall, also called sensitivity or the true positive rate, answers that question. Divide true positives by all actual positives: TP / (TP + FN). A false negative is a positive case the classifier missed. Specificity asks the complementary question about correctly identifying negative cases. section 5.5.5 section 5.5.7
To check a keyword search, first decide what counts as positive. One workable reference is a review score of 4 or 5. Compare that score-based label with the keyword-based label for each review, count true positives and false negatives, then calculate recall. Inspect the underlying reviews too: the score is a chosen reference point, not a substitute for checking whether the labels make sense. section 5.5.7 section 6.2.3
Read recall alongside the mistakes the classifier makes:
- High recall alone is not enough. Labeling every review positive catches every positive review but fails to distinguish negative ones; check specificity as well. section 5.5.5
- Choose an acceptable balance for the decision at hand. Missing a positive review and missing a positive cancer-screening result have very different consequences. section 5.5.5
- If recall is low, investigate the matching process. A Portuguese sentiment lexicon built from lemmas may miss inflected words unless reviews are lemmatized too. section 5.6.4 section 5.6.5
Comparing positive-review methods
In a test of 500 non-empty reviews, a basic keyword search had recall of 0.45 and specificity of 0.90. A ChatGPT 4 API approach reached recall of 0.74 and specificity of 0.93 against the review-score benchmark. That improvement does not make the workflow automatic: the API approach classified model responses with a basic word search that could miss nuanced answers, and API use incurs token costs. section 6.2.3
Table 6.3 Quality of sentiment analysis with different pretrained models
|
Sentiment analysis (
n=500)
|
Number of reviews classified as positive
|
Sensitivity
|
Specificity
|
|---|---|---|---|
|
Reference review score
|
320 (63%)
true positives
|
1
|
1
|
|
Basic keyword search
|
163 (33%)
|
0.45
|
0.9
|
|
ChatGPT 4 API
|
250 (50%)
|
0.74
|
0.93
|
|
Suboptimal model
|
193 (39%)
|
0.56
|
0.93
|
|
Translated inputs
|
295 (59%)
|
0.86
|
0.89
|
|
Multilingual model
|
265 (53%)
|
0.79
|
0.93
|
|
Zero-shot learning model
|
325 (65%)
|
0.87
|
0.74
|
Compare recall (labeled sensitivity) with specificity across keyword search, API classification, translated inputs, multilingual models, and zero-shot learning. No single sensitivity score tells the whole story. section 6.2.9
