liveBook · Manning

Classification score

As explained in Grokking Machine Learning, Second Edition

A real-valued classifier output, often computed as a weighted combination of input features, that represents support for a class before conversion to a prediction or probability.

A classification score is a number that summarizes how strongly a classifier supports one of two classes. Start with a tiny sentiment task: count two words in a sentence, multiply each count by its word weight, and add the results. A nonnegative score can mean happy; a negative score can mean sad. section 5.2 section 5.1

For a perceptron, the score combines every feature with its weight, then adds a bias: weight × feature for each feature, all summed, plus bias. The score of zero is the decision boundary. Scores on opposite sides of zero fall into positive and negative zones, which appear as regions separated by a line. section 5.27 section 5.7

A score with a bias

Suppose crack and doink each have weight 1, while the bias is −3.5. A sentence containing crack once and doink once has score 1 + 1 − 3.5 = −1.5, so it is classified as sad. The bias plays the same role as a threshold: it sets how much positive evidence is needed before the score reaches zero. section 5.4 section 5.5

A score is more informative than a final yes/no label. Two classifiers can make the same wrong prediction, yet the one with a much larger score may be making a more serious mistake because the point lies farther into the wrong zone. Perceptron error uses the absolute score for a misclassified point and zero for a correct one. section 5.12 section 5.19

The same weighted sum also underlies logistic classification. Instead of applying a step function directly, logistic regression applies the sigmoid function to the score. This turns negative and positive scores into values between 0 and 1: zero becomes 0.5, large negative scores approach 0, and large positive scores approach 1. section 6.2 section 6.11

Table 5.4 Our dataset of four sentences with their labels. For each of the two classifiers, we have the score and the prediction.
Sentence (xaack, xbeep)
Label
y
Classifier 1 score
1xaack + 2xbeep – 4
Classifier 1 prediction
step(1xaack + 2xbeep – 4)
Classifier 1 error
Classifier 2 score
–xaack + xbeep
Classifier 2 prediction
step(–xaack + xbeep)
Classifier 2 error
(1,0)
Sad (0)
–3
0 (correct)
0
–1
0 (correct)
0
(0,1)
Happy (1)
-2
0 (incorrect)
2
1
1 (correct)
0
(1,3)
Happy (1)
3
1 (correct)
0
2
1 (correct)
0
(3,2)
Sad (0)
3
1 (incorrect)
3
–1
0 (correct)
0
Mean perceptron error
 
1.25
 
0

Compare two classifiers row by row: each weighted score becomes a prediction, and misclassified rows contribute score-based error. section 5.12

How Grokking Machine Learning, Second Edition covers it

How deep it goes
Derives weighted sums and bias, then implements score, prediction, and score-based perceptron error.
Examples
Uses alien sentences counted by word, two competing classifiers, and a crack-and-doink threshold example.
What it assumes
Teaches scoring from word counts and simple arithmetic before introducing dot products and general feature vectors.
What you can do after
Build it from scratch as a weighted feature sum with bias, then convert it into a perceptron or logistic prediction.
How it connects
Connects perceptrons with weights, bias, decision boundaries, step functions, sigmoid functions, logistic regression, and gradient descent.

Also explained in

Compare how 2 books explain classification score →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning, Second Edition. The section links point to the parts of the book it draws on.