Accuracy is the fraction of evaluated predictions that match their labels. It offers an intuitive score during training, but a high score on familiar examples may not reflect performance on new data. section 4.2.1 section 5.2.5
>>> (1 - (comments.toxic - comments.manual_predict).abs().sum() ... / len(comments)) 0.895...
For toxicity classification, this calculation counts correct predictions and divides by the number of comments. section 4.2.1
Check performance on unseen comments
A toxicity classifier reaches 99.9% accuracy on its training data but 55.4% on held-out test data. The gap warns that training accuracy alone can overstate real-world performance. section 4.2.1
Accuracy also depends on the labels and task: nationality predictions reach 57% on training data and 29% on validation data, while ambiguous name–nationality labels make that comparison difficult to interpret. section 8.2.2
