Chapter 5: Fundamentals of machine learning

Introduction

This chapter introduction previews a framework for understanding overfitting, model evaluation, and the balance between training and generalization.

5.1 Generalization: The goal of machine learning

This section explains why improving performance on training data is not the same as achieving good performance on unseen data.

• Optimization and generalization: Distinguish optimizing a model for performance on its training data from generalizing to data it has never seen, and explain why fitting the training data too well can undermine the latter.
  Why it matters: This distinction keeps you focused on performance beyond the examples used for training.
• Overfitting and separate evaluation: Explain how the chapter 4 examples revealed overfitting and why evaluating only on training data would miss it.
  Why it matters: A rising training score can hide worsening performance on new examples.

Key terms: optimization, generalization, training data, overfitting, training set, validation set, test set, never-before-seen data

5.1.1 Underfitting and overfitting

This section explains how training can move from underfitting to overfitting, why noisy or misleading features make overfitting more likely, and how feature selection can help.

• Underfitting and overfitting pattern: Explain how training and validation performance change as a model moves from underfitting to overfitting.
  Why it matters: The validation curve helps you tell continued useful learning from fitting details that will not transfer to new data.
• Outliers and ambiguous examples: Describe how invalid or mislabeled examples and inherently ambiguous outcomes can lead a model to overfit.
  Why it matters: Not every training example represents a reliable rule that should be applied to future cases.
• Spurious feature correlations: Explain why rare or irrelevant feature values can appear predictive in training data yet hurt generalization, using the MNIST noise-channel comparison.
  Why it matters: A feature can look useful in the training set merely by chance and make predictions worse on new data.
• Informative feature selection: Describe the feature-selection response proposed for features that may be informative or distracting.
  Why it matters: Filtering distracting inputs can prevent a model from relying on chance associations.

Key terms: underfit, overfit, optimization, generalization, validation metrics, noisy training data, outliers, ambiguous features, robust fit, rare features

5.1.2 The nature of generalization in deep learning

This section explains why deep learning can generalize despite being able to memorize arbitrary data: natural data has structure that models can learn to interpolate across.

• Memorization Versus Generalization: Explain why a falling training loss does not by itself show that a model will generalize, using the shuffled-label example.
  Why it matters: It keeps you from mistaking successful fitting for learning a pattern that transfers to new data.
• Natural Data Manifolds: Describe the manifold hypothesis using handwritten digits as an example.
  Why it matters: This structure is the section’s explanation for why learning from a sample of natural data is possible.
• Interpolation and Its Limits: Explain how interpolation supports generalization to unseen inputs and distinguish it from both pixel averaging and human generalization to extreme novelty.
  Why it matters: It explains both how a model handles nearby new examples and why that ability has limits.
• Gradual Manifold Fitting: Explain how gradual training can produce a useful fit before a model memorizes, and why deep learning models are suited to fitting data manifolds.
  Why it matters: It connects the structure of the data to the way training can produce a model that generalizes.
• Data Coverage for Generalization: Explain why the quality and coverage of training data are central to generalization.
  Why it matters: It directs attention to better examples and features, rather than expecting model complexity alone to fix poor generalization.
• Regularization as a Fallback: Describe the section’s proposed alternative when more data is unavailable.
  Why it matters: It gives a reason to constrain a model when improving the training data is not an option.

Key terms: representational power, training loss, validation loss, memorizing, manifold hypothesis, manifold, low-dimensional, input space, interpolation, latent manifold

5.2 Evaluating machine-learning models

This introduction explains why reliable evaluation is necessary when developing models that generalize to new data.

• Measuring model generalization: Explain why developing a model that generalizes to new data requires a reliable way to measure its generalization power.
  Why it matters: Evaluation lets you observe the outcome you are trying to improve: performance on new data.

Key terms: generalization power, evaluate machine learning models

5.2.1 Training, validation, and test sets

This section explains why model evaluation needs separate training, validation, and test data, then compares three ways to split and reuse data when evaluation is unstable.

• Three dataset roles: Explain what the training, validation, and test sets are used for, and why test data must remain untouched until final evaluation.
  Why it matters: Keeping the test set separate makes its final result more meaningful as a measure of performance on new data.
• Validation feedback leakage: Explain how repeated tuning against validation performance can overfit the validation set, even without directly training on it.
  Why it matters: A good validation score can be misleading after many rounds of model selection.
• Hold-out split limitations: Describe simple hold-out validation and explain why its performance estimate may be unreliable with little data.
  Why it matters: A single small hold-out set can give a misleading impression of model performance.
• Repeated K-fold evaluation: Describe how K-fold and iterated shuffled K-fold validation produce scores, and state their trade-off.
  Why it matters: These methods address sensitivity to a particular split, at the cost of more training and evaluation.

Key terms: training data, validation data, test data, hyperparameters, information leaks, overfitting to the validation set, generalization, simple hold-out validation, validation set, test set

5.2.2 Beating a common-sense baseline

This section explains how a common-sense baseline makes validation performance meaningful and helps determine whether a model is worth pursuing.

• Purpose of a trivial baseline: Explain why you should choose a trivial baseline before working with a dataset and compare your model’s validation performance against it.
  Why it matters: A baseline tells you whether an apparently promising validation score is actually progress.
• Choosing a meaningful baseline: Describe simple baselines for classification and explain why class imbalance can make a seemingly high accuracy unimpressive.
  Why it matters: An easy majority-class strategy can set a much higher bar than chance accuracy suggests.

Key terms: common-sense baseline, trivial baseline, validation metrics, random classifier, class imbalance, validation accuracy

5.2.3 Things to keep in mind about model evaluation

This section explains three ways data splits can make model evaluation misleading and why reliable evaluation matters for judging generalization.

• Representative data splits: Explain why training and test sets should represent the data at hand, and why randomly shuffling class-ordered samples before splitting usually prevents a misleading split.
  Why it matters: A model cannot be evaluated meaningfully if its training and test sets contain different parts of the task.
• Time-ordered test data: Explain why future-prediction tasks require a chronological rather than a random training–test split.
  Why it matters: A test on later events is misleading if the model was trained using information from that future.
• Duplicate-free validation splits: Explain why duplicated data points must not occur in both training and validation sets.
  Why it matters: Overlapping examples make validation performance an unreliable measure of performance on unseen data.
• Purpose of reliable evaluation: Explain what reliable model evaluation lets you monitor during machine learning.
  Why it matters: Trustworthy evaluation helps you tell whether improvements in fitting the model carry over beyond the training data.

Key terms: Data representativeness, training set, test set, randomly shuffle, The arrow of time, temporal leak, posterior, Redundancy in your data, validation set, disjoint

5.3 Improving model fit

This section sets the initial model-fitting milestone—beating a simple baseline while becoming capable of overfitting—and identifies three signs that the model has not reached it.

• Initial fit milestone: Explain why the first goal is a model that both generalizes enough to beat a trivial baseline and can overfit, before focusing on reducing overfitting.
  Why it matters: It clarifies why early model development aims for the capacity to overfit rather than avoiding overfitting from the outset.
• Three early fitting problems: Distinguish the three problems at this stage using training loss, validation loss, and performance against a common-sense baseline.
  Why it matters: These signals help you tell which obstacle stands between your current model and the initial fit milestone.

Key terms: generalization power, overfit, common-sense baseline, training loss, validation loss, underfitting

5.3.1 Tuning key gradient descent parameters

This section explains how to get stalled model training moving by tuning the learning rate and batch size.

• Stalled training diagnosis: Explain why a stuck training loss points to the gradient descent configuration rather than an inability to fit the training data.
  Why it matters: A stuck loss is a reason to check training settings before giving up on the model.
• Learning rate and batch size tuning: Describe how changing the learning rate or increasing batch size can help training that has stalled, using the MNIST learning-rate example.
  Why it matters: These are the first adjustments the section recommends for getting training started.

Key terms: loss, gradient descent process, optimizer, initial values, learning rate, batch size, gradients, lower variance

5.3.2 Using better architecture priors

This section explains why a model can fit its training data yet perform no better than chance on validation data, emphasizing the need for suitable architecture priors.

• Insufficient Predictive Information: Explain that fitting the training data does not imply generalization if the inputs lack enough information to predict the targets, as illustrated by shuffled MNIST labels.
  Why it matters: Before changing the model, check whether the data can support the prediction at all.
• Suitable Architecture Priors: Explain that a model may fail to generalize because its architecture makes unsuitable assumptions for the task, and that choosing an architecture with better-suited priors can help.
  Why it matters: When training succeeds but validation does not, the architecture’s fit to the task is worth reconsidering.

Key terms: input data, targets, generalize, shuffled labels, architecture priors, densely connected architecture, recurrent architecture, generalization

5.3.3 Increasing model capacity

This section shows how loss curves can reveal insufficient model capacity and why increasing capacity can make a model overfit, while too much capacity can make it overfit immediately.

• Insufficient capacity diagnosis: Explain why a model whose validation loss stalls without clearly overfitting may need more representational power, and describe the ways the section suggests increasing it.
  Why it matters: A stalled validation curve may signal that the model needs more capacity before you work on limiting overfitting.
• Excessive capacity signal: Describe how the section says to recognize a model with too much memorization capacity.
  Why it matters: The aim is enough capacity to overfit, not unlimited capacity.

Key terms: model capacity, representational power, architecture priors, validation loss, overparameterized, memorization capacity, overfitting

5.4 Improving generalization

5.4.1 Dataset curation

This section explains why improving generalization starts with choosing and curating suitable data, often before improving the model.

• Dataset suitability for generalization: Explain why deep learning is more likely to generalize when data supports smooth interpolation between samples, and why an overly noisy or fundamentally discrete problem may not be suitable.
  Why it matters: It helps you judge whether better data—or a different approach—is needed before investing in a better model.
• Dataset curation priorities: Describe the dataset improvements recommended here: collect enough data to sample the input–output space densely, reduce labeling errors, address dirty or missing data, and select useful features when many are available. Explain why investing in data collection can outperform an equivalent investment in model development.
  Why it matters: These checks direct effort toward data problems that a more elaborate model may not fix.

Key terms: generalization, latent structure, smoothly interpolate, curve fitting, dataset curation, dense sampling, labeling errors, missing values, feature selection

5.4.2 Feature engineering

The section explains how hand-designed input features can simplify a learning problem and why they remain useful even when deep learning can extract features automatically.

• Feature engineering principle: Explain that feature engineering uses knowledge of the data and the algorithm to apply hardcoded, non-learned transformations before data enters a model, making the problem easier.
  Why it matters: The way you represent data can make a difficult learning task much easier.
• Clock feature progression: Trace how the clock-reading example moves from raw pixels to hand-tip coordinates to hand angles, and how each representation simplifies the task.
  Why it matters: The clock example shows how progressively better features can reduce, or even eliminate, the learning required.
• Continuing value of good features: Distinguish the former dependence of shallow algorithms on hand-designed features from deep learning’s ability to extract features, and explain why good features still help.
  Why it matters: Automatic feature learning does not make thoughtful input design pointless, especially when data or resources are limited.

Key terms: feature engineering, hardcoded (non-learned) transformations, input features, raw pixels, (x, y) coordinates, polar coordinates, angle theta, classical shallow algorithms, deep learning, useful features

5.4.3 Using early stopping

This section explains why deep learning training should stop before a complete fit and how validation metrics can identify and preserve the best model state.

• Stopping before a complete fit: Explain why training a deep learning model to the lowest possible training loss is not the aim, and why stopping at the boundary between underfitting and overfitting can improve generalization.
  Why it matters: It explains why a lower training loss is not necessarily a better stopping point.
• Validation guided model selection: Describe how validation metrics are used to choose a stopping point, contrasting retraining for the best number of epochs with retaining the best model during a run using Keras EarlyStopping.
  Why it matters: Using validation results during training can avoid a redundant retraining run while preserving the best observed model.

Key terms: overparameterization, minimum possible training loss, most generalizable fit, underfit curve, overfit curve, validation metrics, epochs, EarlyStopping callback, best known model state

5.4.4 Regularizing your model

This section explains how reducing model capacity, penalizing large weights, and using dropout can limit overfitting, with validation results guiding the choice.

• Regularization and generalization: Explain why regularization deliberately makes a model less able to fit its training data perfectly, and why its effect must be judged through validation.
  Why it matters: A better training fit is not the goal if performance on new data gets worse.
• Model capacity trade-off: Describe how changing the number of layers or units affects overfitting, and how validation loss guides the choice of model size.
  Why it matters: Model size affects both the ability to learn useful patterns and the tendency to memorize training examples.
• Weight penalties in training: Explain how weight regularization discourages large weights, distinguish L1 from L2 penalties, and describe the observed effect of L2 on overfitting.
  Why it matters: Weight penalties offer a way to curb overfitting without shrinking the network.
• Dropout noise and overfitting: Describe what dropout does to layer outputs during training, why that can reduce overfitting, and how training and test-time outputs are kept consistent.
  Why it matters: Dropout reduces reliance on fragile patterns, and it improved validation loss in the section's example.

Key terms: regularization, generalization, validation, capacity, learnable parameters, validation loss, underfit, overfit, weight regularization, L1 regularization

5.5 Summary