Logistic trick

As explained in Grokking Machine Learning

An iterative logistic-regression parameter-update technique that adjusts weights and bias using prediction error, features, and a learning rate.

How can a classifier become a little better each time it sees a labeled example? The logistic trick updates its boundary using the prediction error. Correctly classified points push the boundary farther away, while incorrectly classified points pull it closer. This makes predictions move toward 1 for label 1 and toward 0 for label 0. section 6.2 section 6.6

Apply the trick to one example at a time:

• Compute the prediction ŷ with the sigmoid of the weighted feature sum plus the bias. section 6.3
• Calculate the error y − ŷ. section 6.2
• Update every weight with wi ← wi + η(y − ŷ)xi and update the bias with b ← b + η(y − ŷ). section 6.3
• Repeat for randomly selected points and track total log loss to monitor improvement. section 6.2 section 6.3

A four-point classifier

A small dataset starts with predictions 0.953, 0.731, 0.119, and 0.119. Applying the trick produces updated weights and bias, including a model with weights approximately 0.47 and 0.10 and bias −0.68. Repeating the updates lowers the plotted error over more epochs. section 6.2 section 6.3

Figure 6.10 Left: A plot of all the intermediate steps of the logistic regression algorithm. Notice that we start with a bad classifier and slowly move toward a good one (the thick line). Right: The error plot. Notice that the more epochs we run the logistic regression algorithm, the lower the error gets.

Use the training-path and error plots to see a poor boundary gradually move toward a better classifier. section 6.3

How Grokking Machine Learning covers it

How deep it goes

Explains the geometric rule, update equations, log-loss monitoring, and a hand-coded algorithm.

Examples

Uses a four-point dataset, prediction and loss tables, boundary plots, Python code, and exercises with a chosen learning rate.

What it assumes

Assumes the learner can work with features, labels, weights, bias, and a learning rate while introducing the update procedure.

What you can do after

Build it from scratch by coding predictions, log loss, single-point updates, and repeated logistic-regression training.

How it connects

Connects logistic regression with sigmoid function, log loss, linear classifiers, and the perceptron trick.

liveBookmanning.com

Read Grokking Machine Learning on liveBook Get the book at manning.com

Related concepts

• logistic regressionused by · Logistic regression is a statistical classification model that estimates categorical outcome probabilities by applying a logistic or softmax function to a linear combination of input features.
• cross entropy lossaddresses · Cross-entropy loss measures the expected negative log probability that a predicted distribution assigns to outcomes drawn from a target distribution.
• gradient descentuses · Gradient descent is an iterative optimization method that updates parameters opposite the gradient of an objective function to reduce its value.
• learning rateuses · A learning rate is a step-size parameter that controls the magnitude of updates made by an iterative learning or optimization process.

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning. The section links point to the parts of the book it draws on.