liveBook · Manning

Perceptron algorithm

As explained in Grokking Machine Learning, Second Edition

An iterative training procedure that updates a perceptron's weights and bias using labeled examples.

The perceptron algorithm trains a binary classifier by repeatedly adjusting its weights and bias from labeled examples. Think of the classifier as a line: it should place one class on one side and the other class on the other side. This makes it useful for tasks such as deciding whether an email is spam, a movie will be watched, or a sentence is happy or sad. section 5.0 section 5.27

Start with a random classifier. Pick a labeled point, make a prediction, and check it. If the point is correct, change nothing. If it is wrong, move the line a little toward that point. In algebraic terms, adjust the weights and bias by a small amount to reduce the point’s error. Repeating this small correction many times can produce a separator that classifies most points well. section 5.14 section 5.15

The training loop has a few important moving parts:

  • A labeled dataset supplies features and binary labels, 1 and 0. section 5.15
  • A learning rate controls how far weights and bias move after a mistake. section 5.15
  • Epochs control how many update attempts are made. section 5.15
  • Mean perceptron error tracks progress and can help decide when to stop. section 5.13 section 5.20

A tiny sentiment classifier

Represent each sentence by counts of the words “aack” and “beep.” A line such as #aack − #beep = 0 treats sentences with at least as many “aack” words as happy, and the others as sad. Training shifts the line when a labeled sentence lands on the wrong side. section 5.2 section 5.15

Figure 5.23 If we apply the perceptron trick many times, each time picking a random point, we can imagine that we’ll obtain a classifier that classifies most points correctly.

Follow the repeated updates visually: correctly classified points leave the line still, while mistakes pull it closer until the split improves. section 5.15

A simple implementation calculates a score with a dot product plus bias, converts that score to 0 or 1 with a step function, and updates one randomly selected point at a time. Because point selection is random, separate runs can return different weights and bias; a fixed random seed makes a run repeatable. section 5.19 section 5.20

The method aims for a good separator, not necessarily the best possible one. When two classes cannot be separated by a line, it still seeks a line that separates them as well as possible. The same improvement idea can also be viewed as minimizing mean perceptron error with gradient descent. section 5.15 section 5.16

How Grokking Machine Learning, Second Edition covers it

How deep it goes
Develops repeated updates geometrically and algebraically, then implements score, prediction, error, and training functions.
Examples
Uses happy and sad word-count sentences, COVID-19 symptoms, and AND, OR, and XOR gate datasets.
What it assumes
Starts from classification intuition and introduces perceptrons without assuming prior machine learning knowledge.
What you can do after
Build it from scratch by iterating weight and bias updates, monitoring error, and interpreting a resulting decision line.
How it connects
Connects classification, linear regression, error functions, gradient descent, stochastic gradient descent, and neural networks.

Also explained in

Compare how 2 books explain perceptron algorithm →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning, Second Edition. The section links point to the parts of the book it draws on.