Perceptron error function

As explained in Grokking Machine Learning

An error metric for a perceptron that assigns zero to correctly classified points and the absolute value of the perceptron's score to misclassified points.

Suppose two lines both put a point on the wrong side. Is the mistake equally bad if the point is barely across one line but far across the other? Counting mistakes says yes. The perceptron error function gives the farther-off mistake a larger value. section 5.2

Figure 5.14 The two classifiers misclassify the point. However, the classifier on the right made a much bigger mistake than the classifier on the left. The point on the left is not far from the boundary, and thus, it is not very far from the sad zone. However, the point on the right is very far from the sad zone. Ideally, we would like an error function that assigns a higher error to the classifier in the right than to the classifier in the left.

The two classifiers both misclassify a point, but one places it much farther from the boundary. The error function is meant to distinguish those mistakes. section 5.2

A perceptron calculates a score from a point’s features. A score of zero lies on its boundary; positive and negative scores lie on opposite sides. The score’s absolute value grows as a point moves farther from that boundary. Using the score avoids calculating perpendicular distance to the line. section 5.2

For a point with a known label:

• Predict its label from the score: predict 1 when the score is at least zero, otherwise predict 0. section 5.4
• If the prediction is correct, assign error 0. Otherwise, assign the absolute value of the score: error = |score|. section 5.4
• Average the pointwise errors to obtain the mean perceptron error for a dataset. section 5.4

The mean error helps compare classifiers and track training. In one four-point comparison, the mean errors are 1.25 and 0; the second classifier has the lower error. During training, the perceptron trick repeatedly adjusts the classifier, while the mean error tracks its performance. section 5.2 section 5.3 section 5.4

Try a single update

Take the model with score 2x₁ + 3x₂ − 4 and a point (1, 1) labeled 0. Its score is 1, so it is predicted as 1 and has error 1. With learning rate 0.01, the perceptron trick changes the weights to 1.99 and 2.99 and the bias to −4.01. The new score is 0.97: the prediction is still wrong, but its error has fallen to 0.97. Try calculating the change yourself before checking the numbers. section 5.7 section 5.4

How Grokking Machine Learning covers it

How deep it goes

Defines pointwise error, averages it across a dataset, and implements both; substitutes score magnitude for a perpendicular-distance calculation.

Examples

Compares classifiers on four labeled sentences and tests an error-reducing update on a misclassified point.

What it assumes

Assumes familiarity with binary labels and perceptron scores; defines the error rule before implementing it.

What you can do after

Build it from scratch with a pointwise error function and dataset average, then track that average during perceptron training.

How it connects

Links linear classifiers, scores, the step function, mean error, and the perceptron algorithm.

liveBookmanning.com

Read Grokking Machine Learning on liveBook Get the book at manning.com

Related concepts

• classification scoreuses · A classification score is a real-valued output representing a classifier’s support, confidence, or signed strength for a predicted class.
• objective functionkind of · An objective function assigns values to candidate solutions, predictions, or parameter settings according to a criterion to be minimized or maximized.
• classification error functionused by · An error metric for a two-boundary classifier equal to the sum of the perceptron errors from its two boundary lines.
• perceptron algorithmmeasured by · The perceptron algorithm is an iterative linear-classification procedure that updates model parameters when labeled examples are misclassified.
• datasetmeasures · A dataset is an organized collection of related data items, records, or observations treated as a unit for storage, processing, analysis, or computation.

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning. The section links point to the parts of the book it draws on.