Chapter 8: Yes, no, or maybe so? Logistic regression and classification

Introduction

The introduction distinguishes classification from predicting continuous values and introduces logistic regression as a way to predict a label using a probability.

• Classification versus continuous prediction: Distinguish a classification problem from a continuous-value prediction problem, using examples such as spam labels and temperature. Explain why recognizing the kind of outcome helps you choose an appropriate tool.
  Why it matters: Identifying whether you need a label or a number helps you choose the right kind of model.
• Logistic regression label prediction: Explain that logistic regression predicts a probability for a label and that a sufficiently high probability can be used to predict that label.
  Why it matters: A logistic regression prediction involves a probability before it becomes a chosen label.

Key terms: classification, categorical, binary classification, linear regression, continuous value, logistic regression, probability, label

8.1 Logistic regression

Introduces logistic regression as a way to predict the probability of a classification from an input variable.

• Classification probability from an input: Explain that logistic regression predicts the probability of a label or classification given an input variable, such as the probability of a toddler melting down given hours without a nap.
  Why it matters: This establishes what logistic regression is being used to predict before the chapter develops it further.

Key terms: logistic regression, probability, label/classification, input variable

8.2 Logistic regression intuition

This section uses a toddler-meltdown example to show how logistic regression turns binary observations into predicted probabilities and, with a threshold, yes-or-no predictions.

• Binary observations and predicted probabilities: Distinguish the observed meltdown outcomes from the probabilities the model predicts, and identify hours without a nap as the input in this example.
  Why it matters: It keeps the recorded yes-or-no events separate from the model’s estimates of how likely an event is.
• S-shaped probability curve: Explain why the section fits a logistic S-curve rather than a straight line, and describe how its predicted probability changes as time without a nap increases.
  Why it matters: The curve makes the model’s output interpretable as a changing probability rather than a line drawn through yes-or-no observations.
• Probability threshold for classification: Apply a probability threshold to turn a logistic-regression prediction into a yes-or-no prediction, using the section’s worked example.
  Why it matters: It explains how a probability becomes an actionable classification while remaining an uncertain prediction.
• Toddler model sampling limits: Describe whose behavior the model represents and how the observation schedule limits its application.
  Why it matters: A useful prediction still has to be interpreted within the people and circumstances actually observed.

Key terms: independent variable, output variable, binary, probability, linear regression, S-shaped curve, sigmoid function, logistic function, inflection point, probability threshold

8.3 Logistic regression in Python

This section shows how to fit a logistic regression in Python, interpret its S-shaped probability curve, and use the fitted model to predict a new case.

• Logistic Function And Coefficients: Describe the logistic function’s S-shaped output between 0 and 1, and explain how the input x and coefficients β0 and β1 define the curve in the toddler meltdown example.
  Why it matters: The curve and its coefficients connect the input data to a predicted probability.
• Maximum Likelihood Curve Fitting: Distinguish fitting this logistic curve from fitting a linear regression curve: both find coefficients from x and y data, but logistic regression uses maximum likelihood rather than least squares.
  Why it matters: It explains what the library is doing when it fits the curve, without requiring the fitting mathematics.
• Scikit Learn Fitting Setup: Describe how the example prepares its one input variable for scikit-learn and fits a LogisticRegression model, including what setting penalty to None means here.
  Why it matters: The input shape and fitting settings explain how the example runs and what simplification it makes.
• Coefficients And New Predictions: Trace how the fitted intercept and slope are retrieved and used to predict the probability of a meltdown for a new value of hours without a nap.
  Why it matters: A fitted curve becomes useful when you can obtain a probability for a new case without assuming that fit guarantees future accuracy.
• Multiple Category Extension: Describe the two approaches the note gives for extending logistic regression beyond binary outcomes.
  Why it matters: Binary outcomes are the chapter’s focus, but they are not the only categories logistic regression can address.

Key terms: logistic function, sigmoid function, β0, β1, x, maximum likelihood estimation, least squares, fit, scikit-learn, LogisticRegression

8.4 Understanding the log-odds

This section shows how log-odds make logistic regression’s linear component interpretable and how an odds ratio compares predicted risks at two input values.

• Log-odds and linear component: Explain how to convert a predicted probability to odds and then to log-odds, and describe why the result matches logistic regression’s linear expression for the same input.
  Why it matters: It gives the straight-line part of a logistic model a meaning in terms of the event’s odds.
• Log-odds zero and signs: Describe what positive, negative, and zero log-odds say about an event, and explain why this scale is easier to compare than raw odds.
  Why it matters: The zero point makes it easier to tell which outcome a prediction favors.
• Odds ratio risk comparison: Apply an odds ratio to compare the toddler’s predicted meltdown odds at four and six hours, and distinguish that comparison from comparing probabilities or subtracting log-odds.
  Why it matters: An odds ratio turns two model predictions into a concrete comparison for the nap decision.

Key terms: odds, log-odds, logit function, linear component, 1:1, odds ratio

8.5 Verifying classification models

The section explains why a fitted logistic regression model must be checked on unseen data and why it uses a machine-learning approach to do so.

• Unseen-data model verification: Explain why fitting a logistic regression model is not enough to show that it predicts well, and how withholding data for later validation helps check whether it generalizes.
  Why it matters: A good fit to existing data does not by itself show that a model will make useful predictions.
• Machine-learning approach rationale: Explain why the section favors checking logistic regression predictions on withheld data rather than using the correlation coefficients and p-values discussed for linear regression.
  Why it matters: It explains the section’s choice of validation approach without implying that statistical tests for logistic regression are impossible.

Key terms: generalization, overfitting, train/test splits, cross-validation, validation, machine-learning approach, correlation coefficients, p-values, McFadden’s pseudo-R2, 𝜒2 distribution

8.6 Train/test splits

The section shows how to test a classification model on held-out data and use repeated tests to assess how consistently it predicts unseen cases.

• Train and test datasets: Explain how a train/test split estimates performance on new data: fit the model on one portion and evaluate its predictions on a portion withheld from training.
  Why it matters: A score on data used to build a model does not answer how it will perform on unseen cases.
• Performance across multiple splits: Describe how k-fold cross-validation and repeated random train/test splits give multiple test scores, and explain why their average and spread are useful.
  Why it matters: Several test scores show whether a result is reasonably consistent or depends heavily on one split.
• Prediction versus model explanation: Distinguish checking performance on unseen data from explaining a model's internal workings, and describe the role statistics still plays in examining test results.
  Why it matters: A model can be evaluated for predictive usefulness even when explaining its internal behavior is difficult.
• Limits of classification accuracy: State how the example calculates test accuracy and why the section cautions against declaring a classification model good from that score alone.
  Why it matters: A high percentage correct can sound persuasive without settling whether a classifier is useful.

Key terms: train/test split, training dataset, test dataset, train_test_split(), fit(), predict(), k-fold cross-validation, three-fold cross-validation, standard deviation, unseen data

8.7 False positives and false negatives

This section uses a wolf alarm to explain the four possible classification outcomes and why the costs of its two kinds of error matter.

• Four alarm classification outcomes: Distinguish false positives, false negatives, true positives, and true negatives by saying whether the alarm sounds and whether a wolf is present.
  Why it matters: These labels let you say exactly how an alarm was right or wrong, rather than describing only its overall accuracy.
• Relative costs of alarm errors: Justify why the sheep owner might tolerate more false alarms than missed alarms, while explaining why neither error is desirable.
  Why it matters: The consequences of each mistake help determine which kind of alarm error is more tolerable.

Key terms: false positive, FP, Type I error, false negative, FN, Type II error, false alarms, missed alarms, false positives, false negatives

8.8 Confusion matrices: Why accuracy is misleading

The section uses a confusion matrix to show why 98% accuracy can conceal a wolf detector that never detects a wolf.

• Confusion matrix predictions: Describe how a confusion matrix compares predicted and actual outcomes, and identify true positives, true negatives, false positives, and false negatives in the wolf-detector example.
  Why it matters: The matrix shows not just how often the detector was wrong, but how it was wrong.
• Misleading accuracy under class imbalance: Explain why the vendor’s 98% accuracy did not show that the detector was useful for finding wolves, and compare it with always predicting no wolf.
  Why it matters: A high overall score can hide failure at the very task the detector is meant to perform.

Key terms: confusion matrix, true positives (TP), false positives (FP), true negatives (TN), false negatives (FN), accuracy, class imbalance, true positives, true negatives

8.9 Precision and recall

The section explains how precision and recall reveal a classifier’s success at detecting positive cases, why they can trade off, and why a single combined score may not fit a business decision.

• Precision and recall meanings: Distinguish precision from recall using the wolf alarms: precision asks how often an alarm is right, while recall asks how many actual wolves trigger an alarm. Explain why true negatives are excluded from both.
  Why it matters: The two measures answer different questions about whether the wolf detector is useful.
• Misleading accuracy under class imbalance: Explain how a wolf detector can report 98% accuracy yet be useless at detecting wolves when wolves are uncommon. Interpret zero precision and zero recall in terms of false and missed alarms.
  Why it matters: An impressive accuracy number can hide complete failure on the cases you care about.
• Alarm threshold tradeoff: Explain why raising the threshold for a wolf alarm may reduce false alarms but miss more wolves. Contrast this with always raising an alarm, and describe why the preferred balance depends on the business case.
  Why it matters: Choosing an alarm threshold means deciding which kind of mistake matters more.
• F-measure and business priorities: Describe F1 as the harmonic mean that combines precision and recall into one measure. Explain the section’s objection to giving them equal weight and its preference for choosing which metric matters more to the business.
  Why it matters: A convenient single score should not override the priorities of the decision it supports.

Key terms: precision, recall, true negatives, positive cases, accuracy, class imbalance, false alarms, missed alarms, threshold, F-measure

8.10 Validation in practice

The section shows how to evaluate a classification model across thresholds and choose a threshold that fits the costs of missed events and false alarms.

• Classification threshold tradeoff: Explain how changing the logistic regression threshold changes which predicted probabilities count as meltdowns and shifts precision versus recall.
  Why it matters: A classification decision should reflect what matters more in the situation, not just the default threshold.
• ROC curve interpretation: Describe what the ROC curve plots across thresholds and how its axes, ideal point, and no-skill diagonal help evaluate a classifier.
  Why it matters: The ROC curve lets you inspect the false-alarm cost of catching more positive cases before settling on a threshold.
• ROC inputs and AUC: Explain why an ROC curve uses test-set predicted probabilities rather than fixed predicted classes, and what its AUC summarizes and cannot guarantee.
  Why it matters: A single score is useful for comparison, but it should not be mistaken for proof that a model will work everywhere.
• Imbalanced data and PRC: Explain when the section recommends a precision-recall curve instead of an ROC curve and how to read its threshold tradeoff.
  Why it matters: When one class is rare, the PRC helps you judge the tradeoff most relevant to identifying events of interest.

Key terms: threshold, precision, recall, false positives, false negatives, receiver operating characteristic (ROC) curve, false-positive rate (FPR), true positive rate (TPR), predicted probabilities, predicted classes

8.11 Summary