liveBook · Manning

Logistic regression

As explained in Grokking Machine Learning, Second Edition

A perceptron-like classifier that applies a logistic function to a linear combination of features to produce a continuous class probability or score.

Logistic regression is a classifier that gives a soft answer instead of only yes or no. It combines input features with weights and a bias, then turns that result into a score between 0 and 1. Scores near 1 support one class, scores near 0 support the other, and 0.5 marks the decision boundary. section 6.0 section 6.1 section 6.2

Picture a sentiment model: rather than merely calling a sentence happy or sad, it can rate “I’m good” as somewhat happy and a strongly enthusiastic sentence as much closer to 1. A threshold such as 0.5 can then turn the score into a class label, though the score itself keeps useful information about how strongly the model leans. section 6.0 section 6.1

Figure 6.1 Left: The perceptron algorithm trains a discrete perceptron, where the predictions are 0 (happy) and 1 (sad). Right: The logistic regression algorithm trains a continuous perceptron, where the predictions are numbers between 0 and 1 which indicate the predicted level of happiness.

Compare a discrete perceptron with logistic regression: the same separating boundary can produce either hard labels or gradual scores. section 6.1

The calculation has two small steps. First, multiply each feature by its weight, add the results and the bias, and obtain a linear score. Then apply the sigmoid function. Sigmoid squeezes any real-valued score into the interval from 0 to 1: large negative scores approach 0, zero becomes 0.5, and large positive scores approach 1. section 6.11 section 6.2

A tiny word-count prediction

Suppose features count the words “aack” and “beep,” with weights 1 and 2 and bias −4. For a sentence with one “aack” and two “beep” tokens, the linear score is 1×1 + 2×2 − 4 = 1. Applying sigmoid gives 0.731, so the point falls on the side with scores above 0.5. section 6.3

To fit a logistic classifier to labeled data, the algorithm:

For each update, the prediction error is y − ŷ. Each weight changes by a learning-rate-scaled version of that error times its feature value, while the bias changes by the scaled error. Repeating this process moves the boundary toward misclassified points and farther from correctly classified points; over epochs, log loss can fall sharply. section 6.11 section 6.14

Movie-review sentiment

A practical text classifier can turn reviews into word-count features, map positive and negative sentiments to 1 and 0, and train scikit-learn’s LogisticRegression model. Its learned weights can also identify words associated with more positive or negative sentiment. section 6.12

Logistic regression can use any number of features. With two features, its boundary is a line; with more, it is a higher-dimensional hyperplane. For several classes, multiple linear classifiers can be combined with softmax. In practice, mini-batch gradient descent is used instead of updating from one point at a time. section 6.9 section 6.14

How Grokking Machine Learning, Second Edition covers it

How deep it goes
Works through sigmoid predictions, log loss, iterative updates, and hand-coded training; defers a thorough gradient-descent derivation.
Examples
Uses happy-versus-sad word-count sentences, a small two-feature dataset, and IMDB movie reviews.
What it assumes
Builds on perceptron training and assumes familiarity with its error function and iterative update process.
What you can do after
Build it from scratch with sigmoid, log loss, and weight updates, or apply it through scikit-learn to review sentiment.
How it connects
Connects logistic regression to perceptrons, sigmoid functions, log loss, gradient descent, mini-batch gradient descent, and softmax.

Also explained in

Compare how 16 books explain logistic regression →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning, Second Edition. The section links point to the parts of the book it draws on.