liveBook · Manning

Accuracy

As explained in Grokking Machine Learning, Second Edition

Accuracy is the proportion of evaluated examples for which a model produces the correct prediction.

Classification accuracy answers a simple question: out of all the examples checked, how often did the model give the right label? It is the proportion of correct predictions among all predictions. If a model gets 875 labels right in a test set of 1,000 examples, its accuracy is 875 / 1,000 = 0.875, or 87.5%. section 7.1

This makes accuracy a friendly first measure for comparing classification models. Train several candidates, evaluate each one, and use accuracy as one signal of which model performs best. It is also useful for checking model quality before deployment. section 7.0 section 7.1

To calculate accuracy, follow three small steps:

  • Start with examples whose true labels are known, such as a testing dataset. section 7.1
  • Compare each predicted label with its true label and count the matches. section 7.1
  • Divide the number of matches by the total number of evaluated examples; multiply by 100 for a percentage. section 7.1

A decision stump scored by accuracy

Suppose a decision tree splits six app users into two groups and predicts the most common app label in each group. For a split at age 14.5, both younger users are correct, and two of the four older users are correct. The stump gets 4 of 6 predictions right, so its accuracy is 4 / 6 = 0.667. Comparing possible splits this way can identify the split with the highest accuracy. section 9.9

High accuracy is encouraging, but it is not a guarantee that a classifier is good for the job. Accuracy combines every correct prediction into one number, so it can hide the kinds of mistakes a model makes. This is especially important when positive and negative labels are unbalanced. section 7.0 section 7.18

Look at false positives and false negatives when the two error types have different consequences. A medical diagnosis model may need to avoid false negatives, while a spam-email model may need to avoid false positives. Recall focuses on correctly found positive cases, precision focuses on predicted positives that are truly positive, and F-scores combine those concerns. section 7.18

Accuracy can hide a surprising result

Imagine a disease that affects 1 out of every 10,000 people and a test that is correct 99% of the time for both sick and healthy people. Among one million people, the test correctly identifies 99 sick people but incorrectly labels 9,999 healthy people as sick. Its accuracy sounds strong, yet a positive result corresponds to only 99 sick people out of 10,098 positive results. Accuracy alone does not answer every question about a prediction. section 8.1

Accuracy can appear during training as well. In Keras, adding `metrics=['accuracy']` reports accuracy at each epoch alongside the loss. For an image-recognition model, also calculate testing accuracy by predicting labels for held-out test images and comparing them with their true labels; this helps check whether a good training result is overfitting. section 10.18 section 10.19

Table 9.4 The seven possible questions we can pick, each with the corresponding splitting. In the first set, we put the users who are younger than the cutoff, and in the second set, those who are older than the cutoff.
Question
First set (yes)
Second set (no)
Labels
Weighted accuracy
Weighted Gini impurity index
Weighted entropy
Is the user younger than 7?
empty
12, 14, 15, 25, 32, 35
{},
{A,A,A,C,B,C}
3/6
0.611
1.459
Is the user younger than 13?
12
14, 15, 25, 32, 35
{A},
{A,A,C,B,C}
3/6
0.533
1.268
Is the user younger than 14.5?
12, 14
15, 25, 32, 35
{A,A}
{A,C,B,C}
4/6
0.417
1.0
Is the user younger than 20?
12, 14, 15
25, 32, 35
{A,A,A},
{C,B,C}
5/6
0.222
0.459
Is the user younger than 28.5?
12, 14, 15, 25
32, 35
{A,A,A,C},
{B,C}
4/6
0.416
0.874
Is the user younger than 33.5?
12, 14, 15, 25, 32
35
{A,A,A,C,B},
{C}
4/6
0.467
1.145
Is the user younger than 100?
12, 14, 15, 25, 32, 35
empty
{A,A,A,C,B,C},
{}
3/6
0.611
1/459

Compare the possible age-based questions side by side to see how each split produces a different weighted accuracy. section 9.9

How Grokking Machine Learning, Second Edition covers it

How deep it goes
Defines accuracy, works through numerical calculations, and contrasts its limits with precision and recall.
Examples
Uses a 875-of-1,000 test set, age-based decision stumps, disease testing, neural networks, and handwritten-digit recognition.
What it assumes
Introduces accuracy from zero using known labels and correct predictions.
What you can do after
Evaluate/understand classification accuracy and recognize when error-specific metrics are needed.
How it connects
Connects accuracy to confusion matrices, false positives, false negatives, precision, recall, F-score, sensitivity, and specificity.

Also explained in

Compare how 49 books explain accuracy →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning, Second Edition. The section links point to the parts of the book it draws on.