How often does a review classifier get each rating right—and which mistakes does it make? A confusion matrix compares actual classes with predicted classes. Think of it as a tally sheet: each cell counts one actual–predicted combination. It reveals errors that a single accuracy number can hide. section 3.4.3 section 5.5.7
For review scores from 1 to 5, first prepare the analysis data, split it into training and test sets, fit an ordinal logistic regression model, and predict the test scores. Python’s `confusion_matrix(y_test, y_pred)` then counts the outcomes. Check that the merged DataFrame really exists, contains the intended variables, and has been prepared as the code assumes; an AI-generated workflow may leave those steps unresolved. section 3.4.3
Accuracy: 0.43035067573535907 Confusion Matrix: [[ 385 70 401 1637 1302] [ 38 13 71 563 513] [ 55 16 109 1342 1365] [ 49 18 100 2909 3501] [ 54 23 208 8021 11200]] Coefficients: [-4.92618413e-02 1.63964377e-03 3.08760598e-03 1.38901936e-02 1.93889038e-05 4.21370114e-01 -8.75650964e-06 3.59787432e-01 2.93638614e-02 -1.72296023e-03 2.82130024e-02 9.50451240e-07 1.32936583e-01 2.23209948e-01 1.31639907e-02 -5.95202359e-03]
Inspect the five-by-five output alongside its accuracy of about 0.43. Large counts away from the diagonal show where predicted scores differ from actual scores; the accuracy alone does not show those mistakes. Do not read 43% accuracy as “43% of review-score variability explained.” section 3.4.3 section 4.7
Checking a positive-review keyword search
Suppose a keyword search marks reviews as positive. As a comparison, classify product scores of 4 or 5 as positive. Count true positives, false positives, true negatives, and false negatives, then use those counts to calculate sensitivity and specificity. The score cutoff is a choice, not an unquestionable ground truth; changing it or the keyword list can change the result. section 5.5.7
import pandas as pd
# Assuming you have already loaded the dataframe and defined the
is_positive function
# Define a function to determine if a review is positive based on the
product score
def is_positive_based_on_score(score, threshold=4):
return score >= threshold
# Apply the function to the 'product_score' column
df['is_positive_based_on_score'] =
df['product_score'].apply(is_positive_based_on_score)
# Calculate the confusion matrix
TP = sum((df['is_positive'] == True) & (df['is_positive_based_on_score']
== True))
FP = sum((df['is_positive'] == True) & (df['is_positive_based_on_score']
== False))
TN = sum((df['is_positive'] == False) & (df['is_positive_based_on_score']
== False))
FN = sum((df['is_positive'] == False) & (df['is_positive_based_on_score']
== True))
# Calculate sensitivity (true positive rate) and specificity (true
negative rate)
sensitivity = TP / (TP + FN)
specificity = TN / (TN + FP)
print(f'Sensitivity: {sensitivity:.2f}')
print(f'Specificity: {specificity:.2f}')
The Python example builds the four counts from a keyword flag and a score-based flag. Inspect which flag serves as the comparison before interpreting sensitivity or specificity. section 5.5.7
