Chapter 8: Using convolutions to generalize

Introduction

The introduction explains why the fully connected image classifier struggles to generalize and motivates replacing its dense transformation with convolution.

• Motivation for convolutional layers: Explain why the previous fully connected image classifier tended to memorize training images rather than generalize, and why the chapter proposes convolution as an alternative.
  Why it matters: This explains the problem convolution is introduced to address, without requiring you to know how it works yet.

Key terms: fully connected layers, flattened vector, spatial structure, position independence, convolution

8.1 The case for convolutions

This section motivates convolutions for images by replacing image-wide weighted sums with weighted sums of nearby pixels.

• Nearby pixels and object patterns: Explain why recognizing an object in an image calls for attention to how nearby pixels are arranged, rather than to combinations of pixels far apart.
  Why it matters: This explains why an image model might focus on local neighborhoods instead of mixing every pixel together.
• Local weighted pixel sums: Describe the proposed local computation and contrast it with the image-wide weighted sum of nn.Linear.
  Why it matters: The local weighted sum is the concrete change this section proposes to the earlier fully connected approach.

Key terms: nearby pixels, patterns corresponding to objects, nn.Linear, weighted sum, immediate neighbors, linear operation

8.1.1 What convolutions do

This section explains how convolutions apply a shared, learnable kernel across image neighborhoods to recognize local patterns wherever they occur, using fewer parameters than a fully connected model.

• Local kernel computation: Describe how a convolution produces output values by moving a kernel across an image and taking a weighted sum in each neighborhood, including how an RGB image uses weights for each channel.
  Why it matters: This is the basic computation behind the convolutional layers used later in the chapter.
• Shared weights and location: Explain how reusing one kernel across the image makes a convolution respond to local patterns wherever they occur, and relate that reuse to sparse, tied weights in an equivalent linear operation.
  Why it matters: Sharing local weights is what lets the model look for the same pattern in different parts of an image.
• Convolution parameter savings: Explain why a convolutional layer can have far fewer parameters than the fully connected image model, identifying what determines its parameter count here.
  Why it matters: This explains a major practical advantage of replacing fully connected image operations with convolutions.

Key terms: discrete convolution, kernel, neighborhood, channel, translation invariance, tied weights, backpropagation, parameters, convolution kernel, convolution filters

8.2 Convolutions in action

Introduces PyTorch’s 2D convolution for the bird-versus-airplane images, showing how to configure a layer, interpret its parameters, and inspect its output.

• Conv2d channels and kernel: Describe how to configure nn.Conv2d for the RGB images using input channels, output channels, and kernel size, and explain why multiple output channels are useful.
  Why it matters: These choices determine what patterns the layer can look for and how many kinds of features it can output.
• Convolution parameter shapes: Explain the weight and bias shapes of the example layer in terms of its input channels, output channels, and 3 × 3 filters.
  Why it matters: The parameter shapes show how each output feature combines information from every input color channel.
• Convolution input and output: Describe the tensor shape nn.Conv2d expects for one image and what the example convolution produces, including what can be inferred from its randomly initialized output.
  Why it matters: Reading the input and output shapes helps you run the layer and notice that its spatial dimensions changed.

Key terms: nn.Conv2d, input features, output features, channels, kernel size, conv.weight, bias, out_ch, in_ch, convolutional filters

8.2.1 Padding the boundary

The section explains why convolutions shrink images at their boundaries and how zero padding preserves image size.

• Boundary-driven output shrinkage: Explain why a convolution without padding produces a smaller image, including what happens at a corner and how kernel size determines the output dimensions.
  Why it matters: It explains where the missing border pixels in a convolution’s output went.
• Zero padding for equal size: Describe how padding=1 lets a 3 × 3 convolution produce an output the same size as its input, and distinguish its effect on output size from its effect on weights and bias.
  Why it matters: Padding preserves spatial size without changing the convolution’s parameter sizes.
• Reasons for preserving dimensions: Explain why the section recommends padding convolutions beyond simply avoiding a smaller output.
  Why it matters: Keeping dimensions compatible makes later network structures easier to assemble.

Key terms: convolution kernel, boundary, 3 × 3 neighborhood, padding, ghost pixels, padding=1, kernel_size=3, weight, bias, compatible size

8.2.2 Detecting features with convolutions

Hand-set convolution kernels show how local weighted sums can blur an image or detect edges, before the section explains how a CNN learns such feature-detecting kernels from data.

• Neighborhood averaging and blur: Explain why a constant convolution kernel produces a blurred image and how averaging differs from summing a 3 × 3 neighborhood.
  Why it matters: The blur makes the effect of a kernel’s weights visible in an image.
• Vertical edge detection kernel: Trace how the illustrated kernel responds differently to a vertical intensity boundary and a uniform region.
  Why it matters: It shows that a kernel can highlight a specific image feature rather than merely smooth pixels.
• Learned kernels and feature channels: Explain how a convolutional neural network obtains feature-detecting kernels and what its successive filter banks produce.
  Why it matters: It connects the hand-built blur and edge examples to the filters a convolutional network actually learns.

Key terms: kernel, 3 × 3 neighborhood, blurred, edge-detection kernel, vertical edge, weighted sum, kernels, filter banks, multichannel image, features

8.2.3 Looking further with depth and pooling

This section explains how downsampling between small convolutions lets a network detect larger, more complex image patterns.

• Small kernels and larger patterns: Explain why small convolution kernels alone raise a problem for recognizing larger image structures, and why simply making the kernels image-sized is not the preferred solution.
  Why it matters: It explains why the network needs a way to expand its view without abandoning small convolutions.
• Max pooling and feature survival: Describe how downsampling reduces a feature map’s spatial dimensions and how max pooling chooses which responses survive.
  Why it matters: Pooling shrinks feature maps while retaining strong evidence that a feature was detected nearby.
• Stacked convolutions and receptive fields: Trace how a convolution, downsampling, and another convolution let later outputs depend on a wider input area and combine earlier features.
  Why it matters: Stacking these operations lets small kernels build up to recognizing structures over larger parts of an image.

Key terms: small kernels, locality, translation invariance, fully connected, downsampling, max pooling, feature maps, nn.MaxPool2d, stacking convolutions, receptive field

8.2.4 Putting it all together for our network

Builds a bird-versus-airplane convolutional network from convolutions, pooling, and fully connected layers, then identifies the reshape needed to connect them.

• Convolutional feature pipeline: Trace how the proposed network transforms a 3-channel, 32 × 32 image through two convolution-and-pooling stages into 8-channel, 8 × 8 features.
  Why it matters: Following the shapes shows what the convolutional part passes to the classifier.
• Feature-to-classifier reshape: Explain why the pooled features must become a 512-element vector before the first fully connected layer, and why the shown nn.Sequential model fails without that step.
  Why it matters: The missing reshape explains the linear-layer error rather than treating it as a problem with convolution itself.
• Channel count and capacity: Explain how increasing convolutional output channels would increase the model’s capacity and also enlarge the linear layer.
  Why it matters: Channel counts affect both the features the model can represent and the size of its classifier.

Key terms: nn.Conv2d, nn.MaxPool2d, Tanh, channels, features, 1D vector, nn.Linear(512, 32), nn.Sequential, view, batch dimension

8.3 Subclassing nn.Module

The section explains why and how to subclass nn.Module when a model needs computation beyond a simple sequence of premade layers.

• Flexible model computation: Explain why subclassing nn.Module is useful when premade modules or nn.Sequential do not express the computation a model needs.
  Why it matters: It explains when a custom module is more useful than a sequence of premade layers.
• Forward computation and autograd: Describe the minimum method a subclass must define and how its backward pass is handled when it uses standard torch operations.
  Why it matters: It identifies the code a custom module needs without inventing a manual backward pass.
• Submodule setup and composition: Explain how a custom nn.Module includes other modules and why its constructor calls super().__init__() first.
  Why it matters: It shows how a network can be built from reusable modules while retaining their parameters.

Key terms: nn.Module, nn.Sequential, subclassing, forward, autograd, backward, torch operations, submodules, __init__, self

8.3.1 Our network as an nn.Module

This section rewrites the convolutional classifier as an nn.Module subclass and explains how its intermediate dimensions generally compress image information, with an exception at the first convolution.

• Explicit network forward path: Explain how the Net subclass defines its layers in the constructor and their data flow in forward, including why it reshapes the pooled output with an inferred batch dimension.
  Why it matters: An explicit forward method makes the reshape between convolutional and linear layers possible.
• Classifier information compression: Describe why this classifier’s intermediate values generally shrink, and distinguish its pattern from architectures that reduce spatial resolution while increasing channel count.
  Why it matters: It explains the overall direction of the classifier’s architecture without assuming every dimension must shrink at every layer.
• First convolution dimension exception: Explain why the first convolution is an exception to the network’s general shrinking pattern, while its mapping for one output pixel increases dimension only moderately.
  Why it matters: It qualifies the compression story by separating the size of the whole output from the size of one local mapping.

Key terms: nn.Module, forward, nn.Sequential, self.pool2, view, batch dimension, classification networks, intermediate values, pooling, spatial resolution

8.3.2 How PyTorch keeps track of parameters and submodules

This section explains how PyTorch registers submodules, finds their parameters for optimization, and treats custom module methods differently from standard calls.

• Submodule registration and containers: Explain how assigning an nn.Module to a top-level attribute registers it as a submodule, and why ordinary lists or dictionaries do not provide the same registration.
  Why it matters: Registration determines whether the optimizer can find a model’s trainable parameters.
• Recursive parameter access: Trace how a parent module’s parameters() call reaches nested submodules and makes their gradients available to the optimizer.
  Why it matters: A model can expose its child modules’ trainable parameters without collecting each one by hand.
• Custom method call behavior: Distinguish calling a custom nn.Module method from calling the module through its standard __call__ path.
  Why it matters: A convenient custom method may not trigger behavior attached to a standard model call.

Key terms: nn.Module, submodule, nn.ModuleList, nn.ModuleDict, parameters(), grad, autograd, optimizer, hooks, __call__

8.3.3 The functional API

This section explains PyTorch’s functional API and when to use it instead of modules in a custom network.

• Functional and module state: Distinguish functional calls from nn modules by where their parameters live, and explain why a Module serves as a container for state and forward instructions.
  Why it matters: This distinction makes it easier to see which parts of a network need registered modules.
• API choices in Net: Justify keeping nn.Linear and nn.Conv2d as modules in Net while using functional calls for pooling and activation, and describe when either style may be convenient.
  Why it matters: It helps you simplify a custom forward method without losing track of trainable parameters.

Key terms: functional, torch.nn.functional, nn.Module, Parameters, nn.Linear, nn.Conv2d, pooling, activation, nn.Sequential, functional API

8.4 Training our convolutional neural network

The section assembles the convolutional network’s training loop from a Dataset, DataLoader, model, loss function, and optimizer.

• Convolutional network training loop: Trace how the section sets up and runs training: wrap the Dataset in a DataLoader, create the network, optimizer, and loss function, then process batches within epochs using forward and backward passes and optimizer steps.
  Why it matters: This sequence connects the convolutional model to the training process used in earlier chapters.

Key terms: Dataset, DataLoader, epochs, forward pass, loss, gradients

8.4.1 Measuring accuracy

The section uses training and validation accuracy to evaluate the convolutional classifier and shows that it generalizes better than the earlier fully connected model despite having far fewer parameters.

• Training and validation accuracy: Explain why the section checks accuracy on both the training and validation datasets in addition to loss.
  Why it matters: Accuracy makes the classifier’s performance easier to interpret, especially on images outside its training set.
• Architecture and parameter efficiency: Explain how the convolutional model compares with the fully connected model and what the comparison illustrates.
  Why it matters: A better-suited architecture can improve performance on new images without adding parameters.

Key terms: accuracy, training and validation datasets, loss, fully connected model, convolutional model, locality, translation invariance, generalizing, parameters

8.4.2 Saving and loading our model

The section explains that a saved model file contains parameters, so loading it requires recreating the model structure first.

• Saved parameters and model structure: Explain what the saved birds-versus-airplanes model file contains and how to load it again.
  Why it matters: A parameters-only file cannot recreate the network by itself.

Key terms: parameters, weights and biases, model class

8.4.3 Training on the GPU

This section shows how to train and validate a model on an available GPU and how to handle device placement when loading saved weights.

• GPU device placement for training: Describe how to select a GPU when available, move the model and data-loader tensors to that device, and make the same tensor move in both training and validation.
  Why it matters: Moving only the model or only its inputs does not complete the section’s GPU training pattern.
• Module and tensor move semantics: Distinguish the effects of Module.to and Tensor.to, and explain when to create the optimizer relative to moving the model.
  Why it matters: The two .to methods look alike but have different effects on the objects used in training.
• Saved weight device mapping: Explain why loading saved GPU weights may require a device override and identify map_location as the way shown to provide it.
  Why it matters: A saved model may be loaded on a machine or into a workflow that calls for a different device.

Key terms: GPU, device, torch.cuda.is_available, Tensor.to, nn.Module.to, Module.to, Optimizer, in place, out of place, weights

8.5 Model design

This section explains why more complex data calls for different neural network designs and why PyTorch provides the components to build them.

• Complex image classification demands: Explain why the birds-versus-airplanes classifier is a limited example and why more complex images can require multiple, hierarchically organized visual clues.
  Why it matters: It explains why a network that works on a simple image dataset may need a different design for harder images.
• Architecture and loss choices: Explain how an appropriate architecture and loss function support neural networks on different kinds of data, and describe the PyTorch components available for implementing architectures.
  Why it matters: Choosing how modules connect and how the model is evaluated lets you adapt a network to a new problem rather than reuse one image classifier unchanged.

Key terms: ImageNet, visual clues, hierarchically organized, architecture, loss function, modules, tabular data, sequences, text

8.5.1 Adding memory capacity: Width

The section shows how to widen a convolutional network and explains the resulting trade-off between capacity and overfitting.

• Network width and layer dimensions: Describe network width as neurons per layer or channels per convolution, and explain how to widen this model while keeping later layers and the transition to fully connected layers consistent.
  Why it matters: Changing width affects connected dimensions throughout the model, not just one convolution.
• Capacity and overfitting trade-off: Explain why increasing channel and feature counts raises model capacity and also raises the risk of overfitting.
  Why it matters: A wider model may handle more varied inputs, but its extra parameters are not automatically beneficial.

Key terms: width, output channels, features, view, fully connected layers, parameters, capacity, overfitting, input variability

8.5.2 Helping our model to converge and generalize: Regularization

This section explains how weight penalties, dropout, and batch normalization help a convolutional model train and generalize, including how dropout and batch normalization change at inference time.

• Optimization and generalization: Distinguish decreasing loss on the training set from performing well on unseen data, and explain why regularization is useful.
  Why it matters: A falling training loss alone does not show that the model will work on new images.
• Weight penalties and decay: Explain how L1 and L2 penalties constrain weights, and why L2 regularization is called weight decay. Describe the PyTorch SGD option given for applying it.
  Why it matters: Weight decay offers a built-in way to discourage the model from relying on overly large parameters.
• Dropout during training: Explain what dropout randomly removes during training and why that can reduce overfitting; describe the convolution-specific variant mentioned here.
  Why it matters: Dropout limits a model's opportunity to memorize examples through fixed combinations of neurons.
• Batch normalization effects: Explain how batch normalization shifts and scales intermediate inputs using minibatch statistics, and why the section says this can help training and regularization.
  Why it matters: Batch normalization can ease training while also perturbing samples in a way that helps resist overfitting.
• Training and inference modes: Distinguish how dropout and batch normalization behave during training versus inference, and explain why calling model.eval() matters.
  Why it matters: Switching modes prevents training-time randomness and batch dependence from silently impairing inference.

Key terms: optimization, generalization, regularization, overfitting, L1 regularization, L2 regularization, weight decay, weight_decay, lambda, SGD

8.5.3 Going deeper to learn more complex structures: Depth

This section explains what depth can add to a convolutional network, why deep networks are difficult to train, and how residual blocks and training choices make greater depth practical.

• Depth and hierarchical structure: Explain how greater depth can let a network approximate more complex functions and process hierarchical information through a longer sequence of operations, without claiming deeper is always better.
  Why it matters: This explains why adding layers can be useful before considering the costs of training them.
• Gradient difficulty with depth: Trace why adding many layers can make training fail to converge: backpropagation multiplies derivatives through a long chain, which can leave some parameter contributions to the gradient too small for effective updates.
  Why it matters: This is the training problem that motivates skip connections.
• Residual skip connections: Describe a skip connection as adding a block's input to its output, and explain how the resulting shortcut can give gradients a more direct path and improve convergence.
  Why it matters: The shortcut is the central idea behind the residual networks built in this section.
• Reusable deep residual blocks: Describe how the section constructs a very deep PyTorch network from a reusable residual block and an nn.Sequential collection of blocks, rather than writing each layer separately.
  Why it matters: Reusable blocks make a network with many layers manageable to define and train in PyTorch.
• Deep model convergence choices: Explain why the example adds batch normalization and adjusts initialization and learning rate when training its very deep residual network.
  Why it matters: It prevents the impression that adding skip connections alone guarantees easy training.

Key terms: depth, hierarchical information, backpropagation, vanishing gradients, converge, skip connection, residual networks, ResNets, identity mapping, ResBlock

8.5.4 Comparing the designs from this section

The section compares network design changes while cautioning against reading too much into small accuracy differences and interpreting their training–validation gaps.

• Limits of accuracy comparisons: Explain why the validation accuracy differences among the modified networks should not be treated as decisive, and describe what was held constant in this demonstration.
  Why it matters: A small difference in this comparison is not reliable evidence that one design is better.
• Regularization versus convergence effects: Distinguish the qualitative effect of weight decay and dropout from that of batch norm in the reported training and validation accuracies.
  Why it matters: The training–validation gap helps explain what these changes are doing, even when their validation scores are similar.

Key terms: validation accuracy, random seeds, learning rate, number of epochs, weight decay, dropout, batch norm, regularization, convergence, training accuracy

8.5.5 It’s already outdated

The section argues that, because neural network architectures change quickly, practitioners should focus on turning ideas from papers into PyTorch models or understanding others’ implementations.

• Adapting to changing architectures: Explain why rapidly evolving neural network architectures make it important to translate the math in papers into PyTorch code, or understand code written by others to do so.
  Why it matters: That ability helps you work with new architectures after any particular set of examples becomes dated.

Key terms: neural network architectures, translate the math behind a paper, PyTorch code

8.6 Conclusion

The conclusion describes how to use the bird-or-airplane classifier and highlights two limits that matter for real-world images.

• Classifier input and localization limits: Describe how an incoming image would be prepared for the model and why this classifier cannot by itself pick out a bird or airplane in a larger image.
  Why it matters: Classifying a prepared image is not the same task as finding an object within a larger one.
• Confident predictions on unfamiliar inputs: Explain overgeneralization using the cat example and why it is a problem when real-world inputs cannot be trusted.
  Why it matters: A high-confidence prediction does not guarantee that an unfamiliar input belongs to either class.

Key terms: crop and resize, 32 × 32, bounding boxes, overgeneralization, training distribution

8.7 Exercises

8.8 Summary