Classification accuracy answers a simple question: out of all the examples checked, how often did the model give the right label? It is the proportion of correct predictions among all predictions. If a model gets 875 labels right in a test set of 1,000 examples, its accuracy is 875 / 1,000 = 0.875, or 87.5%. section 7.1
This makes accuracy a friendly first measure for comparing classification models. Train several candidates, evaluate each one, and use accuracy as one signal of which model performs best. It is also useful for checking model quality before deployment. section 7.0 section 7.1
To calculate accuracy, follow three small steps:
- Start with examples whose true labels are known, such as a testing dataset. section 7.1
- Compare each predicted label with its true label and count the matches. section 7.1
- Divide the number of matches by the total number of evaluated examples; multiply by 100 for a percentage. section 7.1
A decision stump scored by accuracy
Suppose a decision tree splits six app users into two groups and predicts the most common app label in each group. For a split at age 14.5, both younger users are correct, and two of the four older users are correct. The stump gets 4 of 6 predictions right, so its accuracy is 4 / 6 = 0.667. Comparing possible splits this way can identify the split with the highest accuracy. section 9.9
High accuracy is encouraging, but it is not a guarantee that a classifier is good for the job. Accuracy combines every correct prediction into one number, so it can hide the kinds of mistakes a model makes. This is especially important when positive and negative labels are unbalanced. section 7.0 section 7.18
Look at false positives and false negatives when the two error types have different consequences. A medical diagnosis model may need to avoid false negatives, while a spam-email model may need to avoid false positives. Recall focuses on correctly found positive cases, precision focuses on predicted positives that are truly positive, and F-scores combine those concerns. section 7.18
Accuracy can hide a surprising result
Imagine a disease that affects 1 out of every 10,000 people and a test that is correct 99% of the time for both sick and healthy people. Among one million people, the test correctly identifies 99 sick people but incorrectly labels 9,999 healthy people as sick. Its accuracy sounds strong, yet a positive result corresponds to only 99 sick people out of 10,098 positive results. Accuracy alone does not answer every question about a prediction. section 8.1
Accuracy can appear during training as well. In Keras, adding `metrics=['accuracy']` reports accuracy at each epoch alongside the loss. For an image-recognition model, also calculate testing accuracy by predicting labels for held-out test images and comparing them with their true labels; this helps check whether a good training result is overfitting. section 10.18 section 10.19
Table 9.4 The seven possible questions we can pick, each with the corresponding splitting. In the first set, we put the users who are younger than the cutoff, and in the second set, those who are older than the cutoff.
|
Question
|
First set (yes)
|
Second set (no)
|
Labels
|
Weighted accuracy
|
Weighted Gini impurity index
|
Weighted entropy
|
|
Is the user younger than 7?
|
empty
|
12, 14, 15, 25, 32, 35
|
{},
{A,A,A,C,B,C}
|
3/6
|
0.611
|
1.459
|
|
Is the user younger than 13?
|
12
|
14, 15, 25, 32, 35
|
{A},
{A,A,C,B,C}
|
3/6
|
0.533
|
1.268
|
|
Is the user younger than 14.5?
|
12, 14
|
15, 25, 32, 35
|
{A,A}
{A,C,B,C}
|
4/6
|
0.417
|
1.0
|
|
Is the user younger than 20?
|
12, 14, 15
|
25, 32, 35
|
{A,A,A},
{C,B,C}
|
5/6
|
0.222
|
0.459
|
|
Is the user younger than 28.5?
|
12, 14, 15, 25
|
32, 35
|
{A,A,A,C},
{B,C}
|
4/6
|
0.416
|
0.874
|
|
Is the user younger than 33.5?
|
12, 14, 15, 25, 32
|
35
|
{A,A,A,C,B},
{C}
|
4/6
|
0.467
|
1.145
|
|
Is the user younger than 100?
|
12, 14, 15, 25, 32, 35
|
empty
|
{A,A,A,C,B,C},
{}
|
3/6
|
0.611
|
1/459
|
Compare the possible age-based questions side by side to see how each split produces a different weighted accuracy. section 9.9
















































