When searching customer reviews for praise, how often does a method correctly leave other reviews unmarked? Specificity answers that question: it is the proportion of actual negative cases classified as negative. Here, “negative” means not positive, so it can include neutral reviews. High specificity means few of those reviews are incorrectly flagged as positive. section 5.5.5 section 6.2.3
To check a positive-review classifier:
- Choose a reference label. One practical rule treats review scores of 4 or 5 as positive and lower scores as not positive. section 5.5.7
- Compare the classifier’s labels with the reference labels. Count true negatives (TN): reference-negative reviews left unmarked. Count false positives (FP): reference-negative reviews marked positive. section 5.5.7
- Calculate specificity as TN / (TN + FP). Check sensitivity too: a method can avoid false positives simply by labeling everything negative, while missing the positives you wanted. section 5.5.5 section 5.5.7
Choosing a review-analysis method
On a sample of 500 reviews, keyword search reached 0.90 specificity and 0.45 sensitivity; a ChatGPT 4 API approach reached 0.93 and 0.74. Both seldom marked reference-negative reviews positive, but the API approach found more reference-positive reviews. Review scores supply the benchmark for this comparison, not an independent reading of every review’s sentiment. section 6.2.3 section 6.2.2
Table 6.3 Quality of sentiment analysis with different pretrained models
|
Sentiment analysis (
n=500)
|
Number of reviews classified as positive
|
Sensitivity
|
Specificity
|
|---|---|---|---|
|
Reference review score
|
320 (63%)
true positives
|
1
|
1
|
|
Basic keyword search
|
163 (33%)
|
0.45
|
0.9
|
|
ChatGPT 4 API
|
250 (50%)
|
0.74
|
0.93
|
|
Suboptimal model
|
193 (39%)
|
0.56
|
0.93
|
|
Translated inputs
|
295 (59%)
|
0.86
|
0.89
|
|
Multilingual model
|
265 (53%)
|
0.79
|
0.93
|
|
Zero-shot learning model
|
325 (65%)
|
0.87
|
0.74
|
Compare specificity alongside sensitivity across keyword search, API classification, translated inputs, and pretrained models; a higher score on one measure need not mean a higher score on the other. section 6.2.9
Set an acceptable specificity for the decision at hand rather than chasing a universal cutoff. If results disappoint, inspect sample reviews and the reference labels, then test changes on the same basis. For example, lemmatizing reviews before dictionary-based classification raised sensitivity from 0.36 to 0.41 while specificity fell from 0.88 to 0.85. section 5.5.5 section 5.6.4 section 5.6.5
