Chapter 5: The mechanics of learning

Introduction

The introduction describes learning in terms of paired examples and the ability to produce outputs for sufficiently similar new inputs.

• Learning from paired examples: Explain that a learning algorithm is given input data paired with desired outputs and, after learning, can produce correct outputs for new data similar enough to its training inputs. Describe how, in deep learning, the input and output can come from different domains, such as an image and a sentence describing it.
  Why it matters: This gives you a concrete description of what learning is meant to achieve before examining how it happens.

Key terms: input data, desired outputs, new data, deep learning

5.1 A timeless lesson in modeling

Kepler’s modeling process illustrates how fitting and validating a parameterized model from data relates to learning, and why the book focuses on adaptable general models.

• Kepler’s model-fitting process: Trace how Kepler used observations to choose an ellipse, estimate its parameters, compare predicted positions with observations, and validate the model on independent data.
  Why it matters: Kepler’s example makes fitting and independent validation concrete before the chapter turns to automated models.
• Learning as parameter fitting: Explain why the section treats fitting data and making an algorithm learn from data as virtually the same process.
  Why it matters: This connects the historical example to what the book means when it says a model learns.
• Specialized and general models: Distinguish Kepler’s task-specific ellipse from the general models the book focuses on, including the tradeoff illustrated by the Newton example.
  Why it matters: It explains why the book moves from Kepler’s hand-chosen ellipse to models that can be trained for different tasks.

Key terms: model, ellipse, parameters, validation, independent observations, fit the data, learn from data, unknown parameters, general models, neural network

5.2 Learning is just parameter estimation

The section sketches how learning estimates model weights by comparing predictions with desired outputs and repeatedly reducing error.

• Model learning cycle: Trace the high-level learning process from input data and desired outputs through prediction, error measurement, weight updates, and evaluation on unseen data.
  Why it matters: This overview gives you a way to connect predictions, loss, and weight updates as the chapter develops them.

Key terms: ground truth, weights, forward pass, loss, gradient, backward pass

5.2.1 A hot problem

The section introduces an unmarked thermometer as an example of a regression problem: fitting model outputs to continuous temperature values.

• Thermometer As Regression Problem: Explain why interpreting the thermometer’s readings is a regression problem: the intended model output is a temperature in familiar units, which is a continuous value.
  Why it matters: Identifying the target as a continuous temperature clarifies what the model is meant to predict.

Key terms: regression problem, continuous values

5.2.2 Gathering some data

The section gathers paired Celsius and new-thermometer readings for estimating what the thermometer’s unknown units mean, while noting that the readings may be noisy.

• Paired temperature measurements: Explain that the data pairs known temperatures in Celsius with readings from the new thermometer in unknown units, so the relationship between them can be worked out. Describe why the measurements may not be exact.
  Why it matters: These paired, imperfect readings are the data used to work out how the new thermometer relates to Celsius.

Key terms: t_c, t_u, noise

5.2.3 Visualizing the data

A quick plot shows noisy data with an apparent pattern, while noting that the example was fabricated to follow a linear model.

• Noisy data and linear pattern: Describe what the plot suggests about the data and distinguish that observation from the authors’ certainty that a linear model is correct.
  Why it matters: It separates what a plot suggests from what is known only because this is a constructed example.

Key terms: noisy, pattern, linear model

5.2.4 Choosing a linear model as a first try

Introduces a linear model for converting unknown temperature readings to Celsius and frames fitting it as minimizing prediction error.

• Linear temperature model: Describe the proposed linear relationship between the unknown temperature readings and Celsius, including what the weight and bias represent.
  Why it matters: This simple model makes clear what its adjustable parameters do.
• Parameter fitting and loss: Explain how the model’s weight and bias are to be chosen from measured temperatures, and what the loss function contributes.
  Why it matters: It connects fitting a line to the objective that guides model training.

Key terms: linear model, weight, bias, t_u, parameters, fitting, loss function, optimization, w, b

5.3 Less loss is what we want

The section defines loss as a numerical measure to minimize and explains why squared error is chosen over absolute error for the temperature model.

• Loss as a learning objective: Explain that a loss function produces one numerical value for the learning process to minimize, typically from differences between model predictions and desired outputs. For the temperature model, the difference is predicted temperature t_p minus measured temperature t_c.
  Why it matters: Loss gives the model-fitting process a concrete measure of improvement.
• Nonnegative error measures: Describe why the raw signed temperature difference must be transformed so errors on either side of the true value count as loss. Identify absolute difference and squared difference as the two examples, both minimized when prediction matches measurement.
  Why it matters: Without treating both directions as errors, the loss would not express the goal of matching the measurement.
• Squared versus absolute loss: Justify the section's choice of squared difference: unlike absolute difference, it has a smooth minimum that is more favorable for optimization, and it penalizes very large errors more heavily. Explain that the choice of loss affects which training-sample errors receive priority.
  Why it matters: The form of the loss affects both how readily it can be minimized and which mistakes the model is pushed to correct.

Key terms: loss function, cost function, t_p, t_c, absolute difference, squared difference, t_p - t_c, smooth, well-defined minimum

5.3.1 From problem back to PyTorch

This section expresses the linear model and mean squared loss in PyTorch, then explains the broadcasting that lets scalar parameters operate on tensor inputs.

• PyTorch model and loss: Describe how the section represents the model and its loss with tensors: the model applies scalar weight and bias parameters to the input tensor, and the loss averages squared differences into one value.
  Why it matters: These tensor operations give the example a model output and a single loss value to reduce.
• Broadcasting across tensor dimensions: Explain why scalar w and b can be used with the input vector, and describe the dimension-matching rules for broadcasting taught here.
  Why it matters: Broadcasting explains how the short model expression applies scalar parameters to every input value.

Key terms: t_u, w, b, zero-dimensional tensors, mean square loss, broadcasting, element wise scaling, element wise offset, index dimension

5.4 Down along the gradient

This section uses a two-knob analogy to explain how gradient descent lowers loss by adjusting parameters and taking smaller steps near a minimum.

• Gradient descent adjustment intuition: Explain how the w and b knob analogy illustrates gradient descent: adjust parameters in directions that reduce loss, avoid steps that overshoot the minimum, and converge through smaller adjustments.
  Why it matters: This intuition makes the later parameter-update steps easier to follow.

Key terms: gradient descent, parameters, loss, w, b, minimum

5.4.1 Decreasing loss

This section turns the idea of adjusting model parameters to reduce loss into a numerical gradient-descent update.

• Numerical loss change estimates: Explain how to estimate the rate of change of loss with respect to a parameter by evaluating loss at values slightly above and below its current value.
  Why it matters: The estimated slope tells you which way a parameter should move to reduce loss.
• Gradient descent parameter updates: Describe how the sign and size of the estimated rate of change guide updates to w and b, and why those updates use a small learning rate.
  Why it matters: This is the basic update step that turns a loss measurement into improved parameter values.

Key terms: rate of change, slope, delta, central difference approximation, gradient descent, learning_rate, parameters, loss

5.4.2 Getting analytical

This section replaces finite-difference estimates with analytical derivatives and combines them into a gradient for the linear model’s weight and bias.

• Limits of numerical gradients: Explain why probing the loss with repeated model evaluations is a poor way to find gradients when a model has many parameters.
  Why it matters: It explains why the section turns from numerical probing to derivatives.
• Analytical derivatives and chain rule: Explain how derivatives provide a gradient for multiple parameters and how the chain rule connects loss, predictions, and model parameters.
  Why it matters: This is the connection that turns a change in each model parameter into a change in loss.
• Per-prediction and parameter gradients: Describe how the section assembles a loss gradient for the linear model from per-prediction loss derivatives and model derivatives.
  Why it matters: It shows how derivatives for individual predictions become a usable gradient for the model’s parameters.

Key terms: delta, neighborhood, loss function, derivative, chain rule, gradient, sum of squares, t_p.size(0), torch.stack, w

5.4.3 Iterating to fit the model

This section explains how repeated parameter updates form a training loop and how the learning rate affects whether that loop converges or diverges.

• Training iterations and epochs: Distinguish an iteration from an epoch, and explain why they are the same in this example but not necessarily with batched data.
  Why it matters: This distinction lets you describe how much data each parameter update has used.
• Learning rate and update stability: Explain why the initial training run diverges and what happens when the learning rate is made much smaller.
  Why it matters: It explains both why training can blow up and why merely making steps tiny is not a complete fix.

Key terms: training loop, epoch, iteration, batch, learning_rate, grad, diverges, converging, hyperparameter

5.4.4 Normalizing inputs

This section rescales temperature inputs to make gradient-descent updates stable, then interprets the fitted parameters and remaining loss.

• Input scaling and gradients: Explain why differently sized gradients make one learning rate difficult to use for both weight and bias, and how rescaling the input addresses that problem in this example.
  Why it matters: Input scaling can make one learning rate work across parameters without tuning a separate rate for each.
• Loss after extended training: Describe what happened to the loss during the longer gradient-descent run and why a nonzero loss does not necessarily mean the updates failed.
  Why it matters: A falling but nonzero loss can reflect imperfect data rather than a broken training process.
• Interpreting the fitted weight: Explain how input normalization affects comparison of the learned weight with the Fahrenheit-to-Celsius formula, and what that comparison suggests about the thermometer.
  Why it matters: Rescaling an input changes how its fitted coefficient should be interpreted in the original units.

Key terms: normalization, gradient, weight, bias, learning rate, t_u, loss, gradient descent, converge, epochs

5.4.5 Visualizing (again)

The section uses a plot to check the fitted linear model against the data and explains the PyTorch tensor argument unpacking used in the plotting code.

• Visual check of model fit: Explain why plotting the data and fitted model is useful, and describe what figure 5.9 suggests about the fit and the measurements.
  Why it matters: A plot can reveal both an apparently good fit and variability in the measurements.
• Tensor argument unpacking: Describe what `*params` does in `model(t_un, *params)` when `params` is a PyTorch tensor.
  Why it matters: It clarifies how the fitted parameters are passed to the model in the plotting code.

Key terms: visualizing our data, linear model, measurements, argument unpacking, leading dimension, params

5.5 PyTorch’s autograd: Backpropagating all things

The section explains how backpropagation obtains loss gradients through differentiable functions and why writing those derivatives by hand becomes cumbersome for deep models.

• Backward propagation of loss gradients: Explain how the chain rule propagates derivatives backward through the loss and model to find the gradient with respect to parameters such as w and b, provided the functions are analytically differentiable.
  Why it matters: This is the gradient calculation behind the parameter updates used in learning.
• Manual derivatives in deep models: Describe why the same gradient calculation applies to a differentiable model with millions of parameters, while writing its analytical derivatives by hand can be tedious and slow for a deep composition of functions.
  Why it matters: It explains why manually deriving gradients becomes inconvenient as models grow deeper.

Key terms: backpropagation, gradient, chain rule, parameters, loss, analytical expression, derivatives, differentiable, linear and nonlinear functions

5.5.1 Computing the gradient automatically

This section shows how PyTorch autograd computes gradients for a model and how to manage those gradients during parameter updates.

• Automatic gradient computation: Explain how setting requires_grad=True on the parameters lets PyTorch compute derivatives of the loss with respect to them when loss.backward() is called.
  Why it matters: Autograd lets you train a model without working out each parameter derivative by hand.
• Gradient accumulation and resetting: Explain why repeated backward calls add to existing gradients and why a training loop must explicitly zero them after using them for parameter updates.
  Why it matters: Failing to clear old gradients silently changes the updates made during training.
• Untracked in-place parameter updates: Justify updating the existing parameters tensor in place inside a torch.no_grad() context.
  Why it matters: Keeping the update outside gradient tracking avoids treating the training step itself as part of the model’s forward computation.

Key terms: autograd, requires_grad, grad, computation graph, backward, leaf nodes, zero_, accumulate, torch.no_grad(), in-place updates

5.5.2 Optimizers à la carte

This section shows how PyTorch optimizers update model parameters in a training loop and how to switch optimization strategies without rewriting that loop.

• Optimizer parameter ownership: Explain that a PyTorch optimizer retains the parameters passed to its constructor so it can read their gradients and update their values.
  Why it matters: This explains how an optimizer can handle even a long list of model parameters.
• Gradient clearing and updates: Trace the roles of zero_grad, backward, and step in a repeatable training loop, including why gradients must be cleared before the next backward call.
  Why it matters: Clearing old gradients keeps each update from unintentionally including gradients from earlier loop iterations.
• Interchangeable optimization strategies: Distinguish the section’s SGD and Adam examples and explain why changing optimizer classes need not change the rest of the training loop.
  Why it matters: You can try a different update strategy without rewriting the training loop.

Key terms: optimizer, parameters, grad attribute, zero_grad, backward, step, gradients, SGD, stochastic gradient descent, Adam

5.5.3 Training, validation, and overfitting

This section explains how independent validation data reveal overfitting, how training and validation losses diagnose model behavior, and how to split data and respond to poor generalization.

• Independent validation and overfitting: Explain why a model needs validation on observations it did not train on and how a flexible model can overfit its training points.
  Why it matters: A low training loss alone cannot tell you whether the model will work on new observations.
• Training and validation loss patterns: Distinguish what training loss and validation loss reveal, including the patterns associated with underfitting and overfitting.
  Why it matters: Reading both losses helps separate a failure to fit from a failure to generalize.
• Ways to improve generalization: Describe the section’s first responses to overfitting and the tradeoff in choosing model capacity.
  Why it matters: A model needs enough flexibility to fit the data, but not so much that it only fits the observed points.
• Paired data splitting and monitoring: Describe how the example creates training and validation sets and what it looks for when evaluating their losses each epoch.
  Why it matters: A consistent split makes the loss comparison meaningful without mismatching inputs and targets.

Key terms: validation set, training set, overfitting, generalizing, training loss, validation loss, underfitting, capacity, regularization, L1

5.5.4 Training set

Explains why a test set must remain independent for final evaluation and how data leakage can make performance estimates misleading.

• Independent final test evaluation: Distinguish the roles of the training, validation, and test sets, and explain why the test set is reserved until the model is fully trained.
  Why it matters: Keeping final evaluation separate helps reveal how the finished model performs without the bias of earlier decisions.
• Data leakage and misleading results: Explain what data leakage is, give the section’s examples, and describe its effect on evaluation and real-world performance.
  Why it matters: A strong evaluation result is unreliable if the model had access to information it would not have in real use.

Key terms: training set, validation set, test set, hyperparameters, overfit, data leakage

5.5.5 Autograd nits and switching it off

This section explains why validation does not affect training gradients and how to avoid unnecessary autograd work during validation.

• Separate training and validation graphs: Explain why evaluating the same model on training and validation data does not cause validation results to influence train_loss.backward().
  Why it matters: It explains why a validation forward pass can coexist with training without changing the training gradient.
• Validation gradient accumulation: Explain what would happen if backward were also called on val_loss.
  Why it matters: It shows why validation loss must remain an independent evaluation rather than another training signal.
• Autograd control during validation: Explain why the validation pass can run without autograd tracking and describe the controls the section gives for doing so.
  Why it matters: Skipping an unused validation graph can save work, especially for large models.
• No-grad output caveat: Distinguish disabling autograd with no_grad from guaranteeing that an output tensor has requires_grad set to False.
  Why it matters: It prevents treating no_grad as a guarantee about every output tensor’s requires_grad attribute.

Key terms: computation graph, train_loss, val_loss, backward, parameters, zero_grad, gradients, torch.no_grad, set_grad_enabled, autograd graph

5.6 Conclusion

The conclusion explains why the chapter used a simple model to make learning through model optimization easier to understand.

• Purpose of the simple model: Explain why the chapter used a simple model for the thermometer problem rather than starting with a neural network.
  Why it matters: The simple example makes the learning process easier to follow before a neural network is introduced.

Key terms: simple model, thermometer problem, neural network

5.7 Exercises

5.8 Summary