Chapter 6: Using a neural network to fit the data

Introduction

The introduction connects the previous chapter’s training process to neural networks and explains why merely choosing a different fixed-form model is not enough for more complex problems.

• Model-independent training process: Explain why the high-level training process from the linear-model example can also be used with a different model architecture.
  Why it matters: It explains why changing the model does not require abandoning the training process already introduced.
• Limits of fixed function shapes: Explain why switching from a linear model to a quadratic model would not address the chapter’s motivation for a different architecture.
  Why it matters: It distinguishes the ability to train a chosen function from the harder problem of choosing an appropriate function shape.

Key terms: backpropagating errors, parameters, gradient, loss, quadratic model, differentiable, function’s shape

6.1 Artificial neurons

Introduces neural networks as compositions of simple functions and defines a neuron as a linear transformation followed by a nonlinear activation.

• Neural networks as compositions: Explain that neural networks can represent complicated functions by composing simpler functions, despite only a slight resemblance to biological neurons.
  Why it matters: This gives you the central idea behind the network before looking at its individual parts.
• Neuron transformation and activation: Describe a neuron’s output as a weighted input plus a bias passed through a fixed nonlinear activation function, using o = f(w × x + b).
  Why it matters: This expression identifies what each part of the neuron contributes to its output.
• Scalar and vector neuron forms: Distinguish the scalar form of the neuron expression from its vector-valued form, which represents a layer of neurons using multidimensional weights and biases.
  Why it matters: It connects the single-neuron expression to a layer containing many neurons.

Key terms: neural networks, composition, neuron, weight, bias, activation function, tanh, layer of neurons, vector-valued, multidimensional weights and biases

6.1.1 Composing a multilayer network

Explains how a multilayer neural network composes layers and uses matrix–vector calculations to represent a whole layer of neurons.

• Multilayer network composition: Explain how layers form a multilayer neural network and why a weight matrix and input vector can represent an entire layer rather than a single weight.
  Why it matters: This is the basic structure behind the multilayer networks built later in the chapter.

Key terms: multilayer neural network, layer of neurons, matrix, vector, linear algebra

6.1.2 Understanding the error function

This section contrasts the error surface of the earlier linear model with that of neural networks, explaining why network parameters must be understood collectively and why nonlinearity helps model real-world data.

• Linear and neural error surfaces: Distinguish the convex error curve of the earlier linear model from the nonconvex error surface of a neural network, even when both use error-squared loss.
  Why it matters: It explains why fitting a neural network is not simply a search for the linear model’s single correct parameter answer.
• Collective meaning of network parameters: Explain why neural-network parameters need not have individually meaningful or uniquely correct values, and why two networks can perform similarly despite different parameter values.
  Why it matters: It directs attention toward the network’s output and performance rather than treating each weight as a separate answer.
• Nonlinearity for real-world patterns: Explain why the section presents neural networks’ nonconvex nature and nonlinear activation functions as useful for real-world problems.
  Why it matters: It connects the difference in error surfaces to the reason for using nonlinear neural networks on complex data.

Key terms: linear model, error-squared loss function, convex error curve, nonconvex, parameters, useful output, individual interpretability, collective system, activation functions, nonlinearity

6.1.3 Adding nonlinearity with activation functions

This section explains why neural networks place nonlinear activation functions after linear operations, both to model varied relationships and to produce useful output ranges.

• Nonlinearity in hidden layers: Explain how an activation function lets a network’s output have different slopes at different values, unlike a linear function alone.
  Why it matters: This is why adding layers of linear operations alone is not the same as building an expressive neural network.
• Meaningful final layer outputs: Explain why an activation at the last layer may constrain or transform a model’s output into values appropriate for its task.
  Why it matters: A score is easier to interpret when the model cannot produce values outside its intended range.
• Capping versus compressing scores: Distinguish hard capping from smooth compression of outputs, and describe why a compressed output responds more to changes near the middle than near its boundaries.
  Why it matters: The shape of an activation determines both the allowed scores and how visibly those scores change.

Key terms: linear operation, nonlinear activation function, different slopes, last layer, output range, torch.nn.Hardtanh, torch.nn.Sigmoid, torch.tanh, sensitive range, boundary values

6.1.4 More activation functions

The section compares several activation functions and explains when ReLU, Sigmoid, and LeakyReLU are useful.

• Activation families and ReLU: Distinguish the smooth activation functions Tanh and Softplus from their hard counterparts, Hardtanh and ReLU, and explain why ReLU receives special attention.
  Why it matters: This comparison gives you a basis for recognizing common activation-function choices.
• Sigmoid output range: Explain why Sigmoid remains useful despite having fallen out of common use.
  Why it matters: It explains why an older activation still has a particular use for outputs.
• LeakyReLU negative-input behavior: Describe how LeakyReLU changes ReLU for negative inputs.
  Why it matters: The negative-input behavior is the distinction between these two related functions.

Key terms: Tanh, Softplus, Hardtanh, ReLU, rectified linear unit, Sigmoid, logistic function, 0…1 range, LeakyReLU, small positive slope

6.1.5 Choosing the best activation function

Explains how an activation function’s nonlinearity and sensitive ranges let networks fit complex functions while determining which units receive useful learning signals.

• Nonlinearity and usable gradients: Explain why activation functions are generally nonlinear and differentiable: nonlinearity prevents stacked linear transformations from collapsing into another affine function, while gradients allow training through the activation.
  Why it matters: These properties explain why adding activations changes what a network can fit without preventing gradient-based training.
• Sensitive and saturated ranges: Distinguish an activation’s sensitive range from a saturated range and describe how each affects the learning signal passed backward.
  Why it matters: The range a unit operates in helps determine whether an error can substantially update it.
• Combined units and complex functions: Describe how units with linear transformations and activations can respond to different input ranges, allowing their combination to approximate complicated functions.
  Why it matters: It connects the behavior of individual activations to the expressive power of a whole network.

Key terms: activation functions, nonlinear, differentiable, affine linear, sensitive range, insensitive, saturated, gradient, backpropagation, linear + activation units

6.1.6 What learning means for a neural network

This section explains how composed neurons can represent nonlinear relationships and how learning tunes their parameters from examples rather than relying on a hand-designed function.

• Nonlinear functions from simple neurons: Explain how composing linear transformations and differentiable activations lets a neural network represent complicated nonlinear input/output relationships. Describe how the four-neuron example produces shapes that individual neurons do not.
  Why it matters: This explains why a network built from simple units can fit relationships much richer than a single unit can.
• Learning parameters for unseen data: Describe training as finding weights and biases that make the network carry out a task, using example inputs and outputs and a loss function for backpropagation. Explain why success includes correct outputs on unseen data from the same data-generating process.
  Why it matters: Fitting the examples is not enough; the learned parameters must also support predictions on new examples.
• Data-driven versus fixed-shape models: Distinguish the thermometer example’s hardcoded linear relationship from a neural network’s flexible family of functions. Explain why learning from examples is useful when a complicated relationship is difficult to specify as an explicit function.
  Why it matters: It clarifies what the network gains—and what explicit explanation it may give up—when the function’s shape is learned from data.

Key terms: linear transformations, differentiable activations, Tanh, neurons, layers, training, weights, biases, loss function, backpropagate

6.2 The PyTorch nn module

Introduces torch.nn modules, how they hold trainable parameters and submodules, and nn.Linear as the built-in counterpart to the earlier linear model.

• Neural Network Module Building Blocks: Explain that torch.nn provides building blocks for neural networks called modules, and that a module is a Python class derived from nn.Module that can hold trainable nn.Parameter tensors.
  Why it matters: This gives you the basic vocabulary for describing a neural network built with PyTorch.
• Submodule Parameter Tracking: Describe how a module tracks parameters in its submodules and why submodules must be top-level attributes or held in nn.ModuleList or nn.ModuleDict.
  Why it matters: How submodules are stored determines whether the optimizer can find their parameters.
• Built In Linear Module: Explain that nn.Linear is an nn.Module subclass that applies an affine transformation using weight and bias, matching the earlier thermometer model.
  Why it matters: It connects the familiar linear model to PyTorch's module-based way of expressing it.

Key terms: torch.nn, module, nn.Module, nn.Parameter, submodules, nn.ModuleList, nn.ModuleDict, optimizer, nn.Linear, affine transformation

6.2.1 Using nn.Module as a callable

Explains how calling an nn.Module runs its forward computation and why user code should not call forward directly.

• Module calls and forward: Explain that an nn.Module instance, such as nn.Linear, can be called like a function: its __call__ method invokes forward with the supplied arguments and performs other work around that invocation. Explain why user code should call the module rather than forward directly.
  Why it matters: Calling the module preserves PyTorch behavior that a direct forward call can skip.

Key terms: nn.Module, __call__, forward, hooks

6.2.2 Returning to the linear model

The section replaces the hand-written linear model with nn.Linear, reshapes inputs for batching, and uses module parameters and a built-in loss in the existing training loop.

• Linear layer features and parameters: Explain what nn.Linear’s input and output feature counts mean for the thermometer model and how many parameters nn.Linear(1, 1) needs.
  Why it matters: Feature counts determine the layer you create and the shape of data you pass to it.
• Batched input and output shapes: Describe the batch-first shape expected by nn modules and how the thermometer inputs are reshaped for nn.Linear.
  Why it matters: The correct batch and feature dimensions let the same module run on many examples at once.
• Reasons for batching inputs: Justify processing multiple inputs as a batch rather than feeding a small model one sample at a time.
  Why it matters: Batching is not just a shape convention; it helps make efficient use of computation.
• Module parameters in optimization: Trace how parameters held by nn.Linear reach the optimizer and are updated during training.
  Why it matters: This explains why the training loop no longer has to pass a separate parameter list into the model.
• Built-in mean squared loss: Describe how nn.MSELoss replaces the hand-written loss function in the training loop and why the results should match.
  Why it matters: A built-in loss can replace the hand-written version without changing what the model is optimizing.

Key terms: nn.Linear, input features, output features, bias, weight, batch, zeroth dimension, B × N_in, unsqueeze, batching

6.3 Finally, a neural network

This section explains why the book moves from a linear model to a neural network using the familiar calibration problem, despite expecting no improvement in model quality.

• Controlled neural network transition: Explain why the calibration example is used to make the transition from a linear model to a neural network, even though the neural network is not expected to produce a higher-quality model.
  Why it matters: The point of this change is to learn the transition, not to improve this particular calibration model.

Key terms: linear model, neural network, approximating function, calibration problem

6.3.1 Replacing the linear model

The section replaces a one-feature linear model with a small neural network and shows how nn.Sequential passes data through its layers.

• Hidden layer and model capacity: Explain the replacement model’s path from one input feature through 13 hidden features and a tanh activation to one output feature, and why multiple hidden units can increase its capacity.
  Why it matters: This explains what changes when the earlier linear model is replaced by a neural network.
• Sequential module execution: Describe how nn.Sequential connects the specified modules and passes data through them in order, while the rest of the fitting setup, including the loss function, stays fixed.
  Why it matters: It connects the network’s layer-by-layer description to the PyTorch model used in place of the linear one.

Key terms: hidden layer, hidden features, tanh activation, linear layer, capacity, nn.Sequential, modules, intermediate outputs, loss function

6.3.2 Inspecting the parameters

This section shows how to inspect a sequential neural network’s trainable parameters and gradients, including how to find parameters by name or through submodules.

• Trainable parameters and gradients: Explain that model.parameters() collects the weights and biases from both linear modules for the optimizer, and that backpropagation populates their gradients before optimizer.step() updates their values.
  Why it matters: It connects the new neural network to the training process you already used for a linear model.
• Parameter-free Tanh activation: Distinguish the Tanh activation from the linear layers when inspecting trainable parameters: Tanh applies a fixed transformation and has no parameters to tune.
  Why it matters: It explains why a layer can appear in the model without contributing trainable parameters.
• Named submodule parameter inspection: Describe how named_parameters() and submodule attributes help identify or inspect particular parameters and gradients, and how naming Sequential modules with an OrderedDict changes their names without changing the sequential data flow.
  Why it matters: Clear parameter names and direct access make it easier to inspect weights or monitor their gradients during training.

Key terms: model.parameters(), grad, loss_train .backward(), optimizer.step(), nn.Tanh, activation functions, parameters, named_parameters, Sequential, OrderedDict

6.3.3 Comparing to the linear model

This section compares the neural network’s fitted output with a line and uses the plot to illustrate overfitting.

• Overfitting in the model comparison: Explain why the neural network’s output departs from a line and what the comparison suggests about its fit to the data.
  Why it matters: A model can fit the data reasonably well overall while still following noise that a simpler line would not.

Key terms: overfit, parameters, measurements

6.4 Conclusion

The section recaps how chapters 5 and 6 built and trained differentiable models, moving from raw autograd to PyTorch’s nn module.

• Model training progression: Describe the progression from building differentiable models and training them with gradient descent using raw autograd to relying on nn.
  Why it matters: This connects the lower-level training approach to the nn-based approach used for neural networks.

Key terms: differentiable models, gradient descent, autograd, nn

6.5 Exercises

6.6 Summary