What does section 10.1.1, Curve smoothing via 1D convolution, cover?

This section shows that a normalized equal-weight 1D convolution kernel performs local averaging to smooth a signal and reveal its broad trend while suppressing short-term noise. Explain how a 1D convolution kernel whose weights are all equal and sum to 1 produces a local average of neighboring input values. Describe the result as a smoothed version of the input curve. Explain why local smoothing is useful: it retains the broad, longer-term trend in an input while reducing short-term fluctuations that are often noise. Relate this effect to low-pass filtering if using frequency-domain language.

What does section 10.1.2, Curve edge detection via 1D convolution, cover?

This section defines edges as sharp changes in a one-dimensional signal, describes an edge-detection convolution’s responses, and explains why edges are informative. Describe an edge in a one-dimensional input array as a sharp change between successive values, and identify the jumps between indices 3 and 4 and between 7 and 8 in the example. Explain that changing convolution-kernel weights changes the convolution’s physical effect, and that the edge-detection filter produces high output responses at the jump locations and low responses elsewhere. Explain why edge locations are useful for image understanding: rapid changes in color or shade provide more semantic clues than flat uniform regions, consistent with evidence about human visual attention.

What does section 10.2, Convolution output size, cover?

This section derives convolution output size from kernel size, input size, stride, and optional zero-padding. Derive the output size for a one-dimensional valid convolution as o = floor((n - k)/s) + 1, where n is input size, k is kernel size, and s is stride. Explain that the formula counts every valid kernel placement, including the initial placement. Trace the boundary argument behind the valid-convolution formula: after m shifts, the kernel's left end is ms and its right end is ms + (k - 1), which must not exceed the last input index n - 1. State that the largest allowable integer m is floor((n - k)/s). State that p zeros added to each side increase a one-dimensional input size from n to n + 2p, giving output size floor((n + 2p - k)/s) + 1. Describe that the same calculation extends to multiple dimensions by applying it separately to each dimension.

What does section 10.3.1, Image smoothing via 2D convolution, cover?

This subsection shows how a uniform 2D convolution kernel smooths a noisy image by replacing each pixel with a local weighted average of its neighboring pixels. Explain how applying 2D convolution with a uniform kernel denoises or smooths an image. State that the operation suppresses local salt-and-pepper fluctuations while making the meaningful image content, such as text for OCR, easier to analyze. Describe the pixel-level effect of the uniform kernel: each output pixel is a weighted local average of the neighboring 3 × 3 input pixels. Explain that this local averaging is what reduces isolated pixel fluctuations.

What does section 10.3.2, Image edge detection via 2D convolution, cover?

This section explains why image edges are semantically useful and how specially chosen 2D convolution kernels detect them by suppressing uniform regions and responding strongly to sharp color changes. Explain why edge pixels carry more useful information for image understanding than pixels in uniform regions, using an example such as a person's silhouette against a plain wall. Describe how a specially chosen 2D convolution kernel detects a local edge: its positive and negative weights cancel on equal-valued pixels but produce a large positive or negative response when they fall on opposite sides of a sharp color jump.

What does section 10.3.3, PyTorch- 2D convolution with custom weights, cover?

This subsection shows how to supply custom 2D convolution kernels in PyTorch and prepare a 2D image for the Conv2D input interface. Describe how fixed uniform and antisymmetric 2D kernels produce local smoothing and edge-detection outputs, respectively, and explain that manually setting such weights is not the usual neural-network workflow in PyTorch. Explain that PyTorch 2D convolution expects an input tensor shaped N × C × H × W, and identify N as batch size, C as channels, H as height, and W as width. Describe how torch.unsqueeze() converts the shown 5 × 5 two-dimensional input tensor into a four-dimensional tensor by adding extra dimensions needed for the batch and channel interface.

What does section 10.4, Three-dimensional convolution, cover?

This section represents video as a spatio-temporal volume and explains why and how a 3D convolutional kernel extracts patterns, including motion, across space and time. Describe a video as a discrete three-dimensional spatio-temporal (ST) volume whose axes are time, height, and width, with each cross-section representing one image frame. Explain why repeated 2D convolution on individual frames cannot fully analyze a video: it loses motion information that is available only by jointly examining successive frames. Trace how a three-dimensional kernel, visualized as a brick, moves through an input ST volume using stride-based slide stops and produces one output value at each stop as a weighted sum of the covered input pixels.

What should I know before reading chapter 10?

Linear layers (chapter 8): The idea that a neural-network layer computes weighted linear combinations of inputs.; The backpropagation algorithm (chapter 8): Backpropagation as the method for propagating loss information and learning network weights.; Dot product of two vectors and its role in machine learning (chapter 2): Multiplying corresponding values and summing products through the dot product.; Array view: Multidimensional arrays of numbers (chapter 2): Arrays as collections of numerical values.; Linear combinations, vector spans, basis vectors, and collinearity preservation (chapter 2): The meaning of a linear combination of vector elements.; Matrix multiplication and machine learning (chapter 2): Matrix multiplication as combinations of input-vector components.; Linear layers expressed as matrix-vector multiplication (chapter 8): The representation of a fully connected linear layer as matrix-vector multiplication.; Training and backpropagation (chapter 8): The training and backpropagation process through which neural-network parameters are adjusted.

Chapter 10: Convolutions in neural networks

What you will learn in chapter 10 of Math and Architectures of Deep Learning

This introduction explains why convolutional layers analyze local image patterns with shared weights and how stacked, trainable convolutions form hierarchical visual features within neural networks.

After this chapter you will be able to

Local convolutional connections. Explain how a convolutional layer produces each output from a small neighborhood of spatially adjacent input values rather than from the entire input. Describe why this makes convolution suitable for detecting local image patterns such as edges, corners, and illumination in pixel neighborhoods.
Shared convolutional weights. Describe shared weights as the same small set of convolution weights sliding across the entire input. Explain that weight sharing both greatly reduces the number of parameters and makes a given convolution extract the same kind of local pattern at every location.
Hierarchical learned features. Trace how stacked convolution layers can build increasingly complex and increasingly global features from local features, from edges and corners in early layers to parts such as eyes, noses, and ears and then faces. Explain that convolution weights are learned through training and can be trained by backpropagation just as fully connected-layer weights can.
Sliding weighted-sum operation. Explain how a 1D convolution produces one output value at each kernel slide stop by multiplying the input elements under the kernel by their corresponding kernel weights and summing the products. Describe how sweeping the kernel from left to right produces the output array.
Convolution size parameters. Describe the roles of input length n, output length o, kernel size k, and stride s in a 1D convolution. Explain that stride is the number of input elements the kernel advances between slide stops, so increasing stride makes the output roughly smaller.
Padding boundary choices. Distinguish valid padding from same (zero) padding when a sliding kernel reaches beyond the input boundary. Explain that valid padding stops before any kernel element falls outside the input, whereas same padding treats outside, ghost input elements as zero and, with stride 1, preserves input-length output.
Equal-weight kernel smoothing. Explain how a 1D convolution kernel whose weights are all equal and sum to 1 produces a local average of neighboring input values. Describe the result as a smoothed version of the input curve.
Trend and noise separation. Explain why local smoothing is useful: it retains the broad, longer-term trend in an input while reducing short-term fluctuations that are often noise. Relate this effect to low-pass filtering if using frequency-domain language.
One-dimensional signal edges. Describe an edge in a one-dimensional input array as a sharp change between successive values, and identify the jumps between indices 3 and 4 and between 7 and 8 in the example.
Edge-detection convolution responses. Explain that changing convolution-kernel weights changes the convolution’s physical effect, and that the edge-detection filter produces high output responses at the jump locations and low responses elsewhere.
Sliding weighted local sums. Explain a stride-1, valid-padding 1D convolution as successive linear combinations of contiguous input elements using the same kernel weights. Describe the output as a moving weighted local sum whose weights can extract local input properties.
Convolution weight-matrix structure. Describe how a 1D convolution can be represented as a weight-matrix times input-vector multiplication. Explain that every matrix row contains the kernel weights in one contiguous shifted location and zeros elsewhere, so each row represents one kernel slide position and produces one output element.
Custom 1D convolution in PyTorch. Describe how to perform a 1D convolution with custom kernel weights using PyTorch's torch.nn.Conv1d class or by directly calling torch.nn.functional.conv1d with input and weight tensors.
Learned versus manual kernel weights. Distinguish the instructional use of manually setting 1D kernel weights from the typical neural-network workflow, where a convolution layer's size, stride, and padding are specified but its weights are learned during training.
Valid convolution output size. Derive the output size for a one-dimensional valid convolution as o = floor((n - k)/s) + 1, where n is input size, k is kernel size, and s is stride. Explain that the formula counts every valid kernel placement, including the initial placement.
Padding and multidimensional sizing. State that p zeros added to each side increase a one-dimensional input size from n to n + 2p, giving output size floor((n + 2p - k)/s) + 1. Describe that the same calculation extends to multiple dimensions by applying it separately to each dimension.
Spatial locality in images. Explain why rasterizing a 2D image into a 1D vector and applying 1D convolution does not correctly extract two-dimensional local patterns. Describe an image as a 2D pixel array whose spatial neighborhoods must be preserved.
Sliding kernel weighted sums. Describe 2D convolution as sliding a 2D kernel over an input image and producing one output value at every slide stop. Explain that this value is the sum of elementwise products between the kernel weights and the image pixels under the kernel.
Two-dimensional convolution parameters. Describe the 2D input, output, kernel, and stride as height-by-width quantities, and explain how stride moves the kernel across rows and then down the image. State that output size is computed separately for height and width, with larger strides reducing the output size.
2D convolution image smoothing. Explain how applying 2D convolution with a uniform kernel denoises or smooths an image. State that the operation suppresses local salt-and-pepper fluctuations while making the meaningful image content, such as text for OCR, easier to analyze.
Uniform kernel local averaging. Describe the pixel-level effect of the uniform kernel: each output pixel is a weighted local average of the neighboring 3 × 3 input pixels. Explain that this local averaging is what reduces isolated pixel fluctuations.
Semantic importance of image edges. Explain why edge pixels carry more useful information for image understanding than pixels in uniform regions, using an example such as a person's silhouette against a plain wall.
Kernel-based local edge detection. Describe how a specially chosen 2D convolution kernel detects a local edge: its positive and negative weights cancel on equal-valued pixels but produce a large positive or negative response when they fall on opposite sides of a sharp color jump.
Custom convolution kernel use. Describe how fixed uniform and antisymmetric 2D kernels produce local smoothing and edge-detection outputs, respectively, and explain that manually setting such weights is not the usual neural-network workflow in PyTorch.
NCHW convolution input layout. Explain that PyTorch 2D convolution expects an input tensor shaped N × C × H × W, and identify N as batch size, C as channels, H as height, and W as width.
Rasterized 2D convolution representation. Describe how a 2D convolution can be represented as matrix multiplication by rasterizing an H × W input image into a vector of length HW and multiplying it by a convolution weight matrix. Explain that each matrix row represents one kernel slide stop and places kernel weights at the vector positions corresponding to that receptive field.
Spatio-temporal video volume. Describe a video as a discrete three-dimensional spatio-temporal (ST) volume whose axes are time, height, and width, with each cross-section representing one image frame.
Need for temporal convolution. Explain why repeated 2D convolution on individual frames cannot fully analyze a video: it loses motion information that is available only by jointly examining successive frames.
3D convolution sweep. Trace how a three-dimensional kernel, visualized as a brick, moves through an input ST volume using stride-based slide stops and produces one output value at each stop as a weighted sum of the covered input pixels.
Temporal motion detection kernel. Explain how a 2 × 3 × 3 3D convolution kernel estimates motion by averaging each 3 × 3 spatial neighborhood in two successive frames and subtracting one temporal average from the other.
Motion response interpretation. Describe why the convolution output is high in regions of motion and low in regions with no motion, and explain why a uniformly colored moving object is detected mainly at its boundary rather than at its center.
3D input tensor layout. Describe PyTorch's N × C × D × H × W layout for a 3D-convolution input, including what batch size, channels, sequence depth, height, and width mean for an image sequence.
3D kernel tensor layout. Explain that PyTorch stores 3D convolution kernels as Cout × Cin × kT × kH × kW, where a bank of filters produces output channels and each filter combines weighted information from its input channels across time and image space.
Transposed convolution interpretation. Explain transposed convolution as multiplying a convolution output by the transpose of the convolution’s weight matrix. Describe how this produces an output-sized vector by distributing output values back to input locations in the same weight proportions used by the forward convolution.
Upsampling versus inversion. Distinguish the upsampling effect of transposed convolution from recovery of the original input. Explain that a valid 1D convolution taking five independent input values to three output values loses information, so the original input cannot in general be recovered from the output.
Embeddings and convolutional encoders. Describe an embedding as a smaller descriptor vector that retains essential features of a class while discarding variable aspects, and explain that an encoder neural network maps an input image to this embedding using convolution layers.
Autoencoder reconstruction training. Explain that an autoencoder pairs an encoder with a decoder: the encoder produces a smaller embedding, and the decoder applies transposed convolutions to reconstruct a tensor the same size as the input. Describe training as minimizing a reconstruction loss between the original and reconstructed input while learning both networks' weights end to end.
Bottleneck feature learning and compression. Explain why an embedding with fewer degrees of freedom than the input prevents perfect reconstruction and forces an autoencoder trained on examples of one class to retain features common to that class. Describe how this can discard background or other variable details and make the embedding a compact compression of the objects of interest.
Transposed convolution output size. Describe how to obtain a transposed convolution output size by inverting equation 10.8, and state the example in which stride 1, valid padding 0, input size 3, and kernel size 3 produce output size 5.
Learned transpose-convolution upsampling. Explain why transpose convolution can be used to upscale a low-resolution input and how it differs from nearest-neighbor, bilinear, or bicubic interpolation. State that interpolation uses a predefined mapping, whereas transpose convolution learns how to distribute input values over a higher-resolution output during training.
Transpose-convolution tensor layouts. State the input and kernel tensor shapes for 2D transpose convolution: the input is N × Cin × H × W and the kernel is Cin × Cout × kH × kW. Distinguish this kernel layout from regular 2D convolution, whose kernel is Cout × Cin × kH × kW.
Learned convolution weights. Explain why convolution weights in a real neural network are learned rather than manually set, contrasting them with the custom weights used earlier for conceptual demonstrations.
Convolution layer setup. Describe what is specified when setting up a convolutional neural network layer: its dimensions and initialized weight values, not its final learned weights.
PyTorch CNN class structure. Describe how a convolutional neural network is structured in PyTorch: it subclasses torch.nn.Module, creates its layers in __init__(), and defines computation in forward().
Pooling and translation invariance. Explain why pooling is used after convolution layers: convolutions are sensitive to feature location, while downsampling makes important features at slightly different positions align more closely in a lower-resolution map. Relate this robustness to local translation invariance.
Local pooling operations. Describe how a pooling layer slides a small filter across a feature map and replaces each local patch with a summary value. Distinguish max pooling, which selects the patch maximum, from average pooling, which computes the patch average.

Cover of Math and Architectures of Deep Learning

liveBook manning.com

Read chapter 10 on liveBook Get the book at manning.com

What it builds on

Linear layers (chapter 8) · The idea that a neural-network layer computes weighted linear combinations of inputs.
The backpropagation algorithm (chapter 8) · Backpropagation as the method for propagating loss information and learning network weights.
Dot product of two vectors and its role in machine learning (chapter 2) · Multiplying corresponding values and summing products through the dot product.
Array view: Multidimensional arrays of numbers (chapter 2) · Arrays as collections of numerical values.
Linear combinations, vector spans, basis vectors, and collinearity preservation (chapter 2) · The meaning of a linear combination of vector elements.
Matrix multiplication and machine learning (chapter 2) · Matrix multiplication as combinations of input-vector components.
Linear layers expressed as matrix-vector multiplication (chapter 8) · The representation of a fully connected linear layer as matrix-vector multiplication.
Training and backpropagation (chapter 8) · The training and backpropagation process through which neural-network parameters are adjusted.

Section by section

Introduction

This introduction explains why convolutional layers analyze local image patterns with shared weights and how stacked, trainable convolutions form hierarchical visual features within neural networks.

Local convolutional connections: Explain how a convolutional layer produces each output from a small neighborhood of spatially adjacent input values rather than from the entire input. Describe why this makes convolution suitable for detecting local image patterns such as edges, corners, and illumination in pixel neighborhoods.
Why it matters: Local connections let a network focus on the nearby visual evidence that defines meaningful image features.
Shared convolutional weights: Describe shared weights as the same small set of convolution weights sliding across the entire input. Explain that weight sharing both greatly reduces the number of parameters and makes a given convolution extract the same kind of local pattern at every location.
Why it matters: Weight sharing makes convolution practical for large images while allowing a learned feature detector to operate everywhere.
Hierarchical learned features: Trace how stacked convolution layers can build increasingly complex and increasingly global features from local features, from edges and corners in early layers to parts such as eyes, noses, and ears and then faces. Explain that convolution weights are learned through training and can be trained by backpropagation just as fully connected-layer weights can.
Why it matters: Learning and stacking convolutional features avoids hand-designing image detectors and enables recognition of complex structures.

Key terms: convolution, convolution layer, local connections, local patterns, spatially adjacent input values, shared weights, convolution operator, fully connected layer, global view, multilayer convolutional neural networks

10.1 One-dimensional convolution: Graphical and algebraical view

This section introduces 1D convolution as a sliding weighted-sum operation and defines the input, kernel, stride, padding, and output positions that determine it.

Sliding weighted-sum operation: Explain how a 1D convolution produces one output value at each kernel slide stop by multiplying the input elements under the kernel by their corresponding kernel weights and summing the products. Describe how sweeping the kernel from left to right produces the output array.
Why it matters: This sliding local computation is the fundamental operation that later convolutional layers apply to signals, images, and videos.
Convolution size parameters: Describe the roles of input length n, output length o, kernel size k, and stride s in a 1D convolution. Explain that stride is the number of input elements the kernel advances between slide stops, so increasing stride makes the output roughly smaller.
Why it matters: These parameters specify where convolution computes outputs and control how much a convolution reduces the resolution of its input.
Padding boundary choices: Distinguish valid padding from same (zero) padding when a sliding kernel reaches beyond the input boundary. Explain that valid padding stops before any kernel element falls outside the input, whereas same padding treats outside, ghost input elements as zero and, with stride 1, preserves input-length output.
Why it matters: Padding determines whether boundary information is discarded and is essential for predicting a convolution layer’s output size.
Stride-defined output grid: Describe the output locations as the kernel-origin slide stops selected from the input grid: starting at zero and advancing by the stride until the right end is reached. State that there is one output value for each such output-grid point.
Why it matters: Thinking of outputs as a stride-selected grid prepares you to generalize convolution from lines to spatial grids.

Key terms: 1D convolution, input array, kernel, kernel weights, slide stop, output array, input, output, n, o

10.1.1 Curve smoothing via 1D convolution

This section shows that a normalized equal-weight 1D convolution kernel performs local averaging to smooth a signal and reveal its broad trend while suppressing short-term noise.

Equal-weight kernel smoothing: Explain how a 1D convolution kernel whose weights are all equal and sum to 1 produces a local average of neighboring input values. Describe the result as a smoothed version of the input curve.
Why it matters: This explains why a convolution can transform noisy local measurements into a more useful signal.
Trend and noise separation: Explain why local smoothing is useful: it retains the broad, longer-term trend in an input while reducing short-term fluctuations that are often noise. Relate this effect to low-pass filtering if using frequency-domain language.
Why it matters: Smoothing is valuable when short-lived variation obscures the underlying pattern that a model or analyst needs to see.

Key terms: 1D kernel, equal weights, normalized, local averaging, smoothing, moving average, broad trend, short-term fluctuations, noise, low-pass filter

10.1.2 Curve edge detection via 1D convolution

This section defines edges as sharp changes in a one-dimensional signal, describes an edge-detection convolution’s responses, and explains why edges are informative.

One-dimensional signal edges: Describe an edge in a one-dimensional input array as a sharp change between successive values, and identify the jumps between indices 3 and 4 and between 7 and 8 in the example.
Why it matters: Edges mark locations where a signal changes abruptly, which often carry more useful information than uniform regions.
Edge-detection convolution responses: Explain that changing convolution-kernel weights changes the convolution’s physical effect, and that the edge-detection filter produces high output responses at the jump locations and low responses elsewhere.
Why it matters: A convolution filter can make specific input patterns, such as abrupt transitions, stand out in its output.
Semantic value of edges: Explain why edge locations are useful for image understanding: rapid changes in color or shade provide more semantic clues than flat uniform regions, consistent with evidence about human visual attention.
Why it matters: Detecting edges helps a model focus on image locations that are likely to reveal meaningful structure.

Key terms: edge, input array, convolution kernel, edge-detection convolution, filter, high response, output value, semantic clues, flat uniform regions, human visual cortex

10.1.3 One-dimensional convolution as matrix multiplication

This section recasts one-dimensional convolution as multiplication by a sparse, structured weight matrix, showing why it can be trained like a linear layer.

Sliding weighted local sums: Explain a stride-1, valid-padding 1D convolution as successive linear combinations of contiguous input elements using the same kernel weights. Describe the output as a moving weighted local sum whose weights can extract local input properties.
Why it matters: This is the basic computation that convolution layers perform before they can learn useful local features.
Convolution weight-matrix structure: Describe how a 1D convolution can be represented as a weight-matrix times input-vector multiplication. Explain that every matrix row contains the kernel weights in one contiguous shifted location and zeros elsewhere, so each row represents one kernel slide position and produces one output element.
Why it matters: The matrix view connects convolution to familiar linear algebra and makes its local, sparse connectivity explicit.
Stride and neural-layer training: Explain how stride appears in the matrix representation and why convolution can use the same forward-propagation and backpropagation framework as a fully connected linear layer. State that a stride of 2 shifts the kernel-weight locations by two columns in successive matrix rows, and that the matrix representation is conceptual rather than the most efficient implementation.
Why it matters: Seeing convolution as a structured linear layer explains both how its parameters are trained and why optimized convolution implementations are needed.

Key terms: convolution, kernel, stride, valid padding, linear combinations, moving weighted local sum, matrix-vector multiplication, weight matrix, sparse, block-diagonal structure

10.1.4 PyTorch- One-dimensional convolution with custom weights

This subsection shows how PyTorch performs one-dimensional convolution with manually chosen kernels and contrasts that demonstration with the usual learned convolution-layer weights.

Custom 1D convolution in PyTorch: Describe how to perform a 1D convolution with custom kernel weights using PyTorch's torch.nn.Conv1d class or by directly calling torch.nn.functional.conv1d with input and weight tensors.
Why it matters: This makes the mathematical operation of convolution concrete before convolution layers are used inside larger neural networks.
Learned versus manual kernel weights: Distinguish the instructional use of manually setting 1D kernel weights from the typical neural-network workflow, where a convolution layer's size, stride, and padding are specified but its weights are learned during training.
Why it matters: The distinction clarifies why the examples expose exact filter values even though trained convolutional networks usually do not require manually chosen weights.

Key terms: torch.nn.Conv1d, torch.nn.functional.conv1d, input tensor, weight tensor, 1D convolution, kernel, weights, convolution layer, stride, padding

10.2 Convolution output size

This section derives convolution output size from kernel size, input size, stride, and optional zero-padding.

Valid convolution output size: Derive the output size for a one-dimensional valid convolution as o = floor((n - k)/s) + 1, where n is input size, k is kernel size, and s is stride. Explain that the formula counts every valid kernel placement, including the initial placement.
Why it matters: Output-size calculations let you design convolution layers whose tensor dimensions fit together.
Boundary-based size derivation: Trace the boundary argument behind the valid-convolution formula: after m shifts, the kernel's left end is ms and its right end is ms + (k - 1), which must not exceed the last input index n - 1. State that the largest allowable integer m is floor((n - k)/s).
Why it matters: The boundary argument helps you reconstruct the size formula rather than memorizing it.
Padding and multidimensional sizing: State that p zeros added to each side increase a one-dimensional input size from n to n + 2p, giving output size floor((n + 2p - k)/s) + 1. Describe that the same calculation extends to multiple dimensions by applying it separately to each dimension.
Why it matters: Padding and per-dimension sizing are essential for predicting the shapes of image and volume convolution outputs.

Key terms: kernel, input size, stride, valid padding, output size, left end, right end, shift, last valid position, zero-padding

10.3 Two-dimensional convolution: Graphical and algebraic view

This section introduces 2D convolution as a spatially local sliding weighted-sum operation on images, defining its kernel, stride, padding, output grid, and algebraic form.

Spatial locality in images: Explain why rasterizing a 2D image into a 1D vector and applying 1D convolution does not correctly extract two-dimensional local patterns. Describe an image as a 2D pixel array whose spatial neighborhoods must be preserved.
Why it matters: Convolution works for images because it operates on nearby pixels in the image plane rather than artificial neighbors created by flattening.
Sliding kernel weighted sums: Describe 2D convolution as sliding a 2D kernel over an input image and producing one output value at every slide stop. Explain that this value is the sum of elementwise products between the kernel weights and the image pixels under the kernel.
Why it matters: This sliding weighted-sum mechanism is the operation through which convolution extracts local patterns from an image.
Two-dimensional convolution parameters: Describe the 2D input, output, kernel, and stride as height-by-width quantities, and explain how stride moves the kernel across rows and then down the image. State that output size is computed separately for height and width, with larger strides reducing the output size.
Why it matters: Kernel size and stride determine which local regions are examined and how densely the output represents the input.
Valid and same padding: Distinguish valid padding from same zero padding in 2D convolution. Explain that valid padding stops when any kernel part would leave the input, whereas same padding continues while the kernel's top-left corner is valid and treats positions outside the image as zeros.
Why it matters: Padding determines whether border pixels can contribute to outputs and strongly affects output dimensions.
Output-grid algebraic formulation: Describe the output grid as the set of stride-determined kernel top-left positions, with one output at each such position. Relate Equation 10.5 to multiplying every pixel in the kernel-covered domain by its corresponding kernel weight and summing the products.
Why it matters: The output-grid and indexed formulation connects the visual sliding-tile intuition to the mathematical definition of convolution.

Key terms: rasterize, two-dimensional neighborhoods, local patterns, pixel, Manhattan distance, 2D convolution, kernel, input array, output array, slide stop

10.3.1 Image smoothing via 2D convolution

This subsection shows how a uniform 2D convolution kernel smooths a noisy image by replacing each pixel with a local weighted average of its neighboring pixels.

2D convolution image smoothing: Explain how applying 2D convolution with a uniform kernel denoises or smooths an image. State that the operation suppresses local salt-and-pepper fluctuations while making the meaningful image content, such as text for OCR, easier to analyze.
Why it matters: Image smoothing can remove visually irrelevant noise before a neural network or image-analysis system processes the meaningful content.
Uniform kernel local averaging: Describe the pixel-level effect of the uniform kernel: each output pixel is a weighted local average of the neighboring 3 × 3 input pixels. Explain that this local averaging is what reduces isolated pixel fluctuations.
Why it matters: Knowing that convolution computes local weighted combinations lets you connect a kernel's weights to the image transformation it produces.

Key terms: 2D convolution, image smoothing, denoising, salt-and-pepper noise, uniform kernel, uniform weights, kernel, output pixel, weighted local average, 3 × 3 input pixels

10.3.2 Image edge detection via 2D convolution

This section explains why image edges are semantically useful and how specially chosen 2D convolution kernels detect them by suppressing uniform regions and responding strongly to sharp color changes.

Semantic importance of image edges: Explain why edge pixels carry more useful information for image understanding than pixels in uniform regions, using an example such as a person's silhouette against a plain wall.
Why it matters: Edges concentrate the visual changes that help a model distinguish meaningful image structure from uninformative background.
Kernel-based local edge detection: Describe how a specially chosen 2D convolution kernel detects a local edge: its positive and negative weights cancel on equal-valued pixels but produce a large positive or negative response when they fall on opposite sides of a sharp color jump.
Why it matters: This explains how convolution turns local changes in an image into feature responses that a neural network can use.

Key terms: semantic importance, edge pixels, silhouette, image understanding, sharp changes in color, 2D convolution, kernel, weighted sum, uniform regions, sharp jump in color

10.3.3 PyTorch- 2D convolution with custom weights

This subsection shows how to supply custom 2D convolution kernels in PyTorch and prepare a 2D image for the Conv2D input interface.

Custom convolution kernel use: Describe how fixed uniform and antisymmetric 2D kernels produce local smoothing and edge-detection outputs, respectively, and explain that manually setting such weights is not the usual neural-network workflow in PyTorch.
Why it matters: This distinguishes demonstrations with hand-designed filters from convolutional layers whose filters are learned during training.
NCHW convolution input layout: Explain that PyTorch 2D convolution expects an input tensor shaped N × C × H × W, and identify N as batch size, C as channels, H as height, and W as width.
Why it matters: Using the required layout lets a 2D image be passed correctly to PyTorch convolution operations.
Image dimension expansion: Describe how torch.unsqueeze() converts the shown 5 × 5 two-dimensional input tensor into a four-dimensional tensor by adding extra dimensions needed for the batch and channel interface.
Why it matters: Dimension expansion is the practical bridge between a plain image array and the tensor shape expected by Conv2D.

Key terms: kernel, uniform weights, antisymmetric weights, local smoothing, edge detection, Conv2D, N × C × H × W, batch size, channels, height

10.3.4 Two-dimensional convolution as matrix multiplication

This section recasts a specific valid 2D convolution as multiplication of a rasterized image vector by a structured weight matrix.

Rasterized 2D convolution representation: Describe how a 2D convolution can be represented as matrix multiplication by rasterizing an H × W input image into a vector of length HW and multiplying it by a convolution weight matrix. Explain that each matrix row represents one kernel slide stop and places kernel weights at the vector positions corresponding to that receptive field.
Why it matters: This representation makes the linear structure of convolution visible and prepares you to understand transposed convolution.
4-by-4 convolution matrix dimensions: Trace the dimensions in the section's example: a 4 × 4 image becomes a 16-element rasterized vector, and valid convolution with a 2 × 2 kernel and stride 1 has nine slide stops. State that the resulting weight matrix has 9 rows and 16 columns and produces nine outputs that can be folded into a 3 × 3 output image.
Why it matters: Checking these dimensions helps you connect convolution windows, output positions, and matrix multiplication correctly.

Key terms: rasterized 1D vector, convolution weight matrix, slide stop, kernel weights, receptive field, valid padding, stride, 2 × 2 kernel, 9 × 16 weight matrix, 3 × 3 convolution output image

10.4 Three-dimensional convolution

This section represents video as a spatio-temporal volume and explains why and how a 3D convolutional kernel extracts patterns, including motion, across space and time.

Spatio-temporal video volume: Describe a video as a discrete three-dimensional spatio-temporal (ST) volume whose axes are time, height, and width, with each cross-section representing one image frame.
Why it matters: This representation makes it possible to treat dynamic video content as structured input to a neural network.
Need for temporal convolution: Explain why repeated 2D convolution on individual frames cannot fully analyze a video: it loses motion information that is available only by jointly examining successive frames.
Why it matters: Many important video patterns, such as whether a door is opening or closing, cannot be inferred from a single frame.
3D convolution sweep: Trace how a three-dimensional kernel, visualized as a brick, moves through an input ST volume using stride-based slide stops and produces one output value at each stop as a weighted sum of the covered input pixels.
Why it matters: This sliding weighted-sum operation is the concrete computation by which a 3D convolution converts local spatio-temporal patterns into an output volume.

Key terms: spatio-temporal volume, ST volume, frame, pixel, 2D convolution, 3D convolution, motion, successive frames, kernel, stride vector

10.4.1 Video motion detection via 3D convolution

This subsection explains how a 3D convolution kernel detects video motion by comparing spatially averaged pixel values across successive frames.

Temporal motion detection kernel: Explain how a 2 × 3 × 3 3D convolution kernel estimates motion by averaging each 3 × 3 spatial neighborhood in two successive frames and subtracting one temporal average from the other.
Why it matters: This shows how convolution can extract changes over time rather than only visual patterns within a single image.
Motion response interpretation: Describe why the convolution output is high in regions of motion and low in regions with no motion, and explain why a uniformly colored moving object is detected mainly at its boundary rather than at its center.
Why it matters: Interpreting the output correctly prevents treating every part of a moving object as equally detectable by a frame-difference filter.

Key terms: 3D convolution, 2 × 3 × 3 kernel, space-time volume, ST volume, spatial neighborhood, motion detector, regions of motion, regions of no motion, boundary of motion

10.4.2 PyTorch- Three-dimensional convolution with custom weights

This section explains the PyTorch tensor and kernel shapes needed to implement a 3D convolution for motion detection with custom weights.

3D input tensor layout: Describe PyTorch's N × C × D × H × W layout for a 3D-convolution input, including what batch size, channels, sequence depth, height, and width mean for an image sequence.
Why it matters: Correct tensor-axis meanings are necessary to supply video or image-sequence data to a PyTorch 3D convolution layer.
3D kernel tensor layout: Explain that PyTorch stores 3D convolution kernels as Cout × Cin × kT × kH × kW, where a bank of filters produces output channels and each filter combines weighted information from its input channels across time and image space.
Why it matters: Kernel dimensions determine how many input channels a layer can accept, how many output feature channels it produces, and the temporal and spatial extent it examines.
Motion detector tensor example: State the concrete tensor and kernel dimensions for the motion-detector example: one sequence of five 320 × 320 grayscale frames has input shape 1 × 1 × 5 × 320 × 320, and its single kernel has shape 1 × 1 × 2 × 3 × 3.
Why it matters: Working through the example connects the abstract PyTorch dimension conventions to a specific motion-detection setup.

Key terms: N × C × D × H × W, batch size, input channels, sequence length, grayscale images, Cout × Cin × kT × kH × kW, output channels, 3D filters, kernel bank, kT

10.5 Transposed convolution or fractionally strided convolution

This section introduces transposed convolution as multiplication by a convolution weight matrix’s transpose, which upsamples an output without reconstructing the original input.

Transposed convolution interpretation: Explain transposed convolution as multiplying a convolution output by the transpose of the convolution’s weight matrix. Describe how this produces an output-sized vector by distributing output values back to input locations in the same weight proportions used by the forward convolution.
Why it matters: This explains why transposed convolution can enlarge a representation while using the learned convolution weights.
Upsampling versus inversion: Distinguish the upsampling effect of transposed convolution from recovery of the original input. Explain that a valid 1D convolution taking five independent input values to three output values loses information, so the original input cannot in general be recovered from the output.
Why it matters: An enlarged transposed-convolution output is a new weighted reconstruction, not evidence that lost convolution information has been restored.
Backpropagation analogy: Describe the analogy between transposed convolution and backpropagation through a linear layer: both multiply by the transpose of the forward weight matrix and distribute quantities backward in proportion to the original forward contributions. State that the transpose-convolution idea extends from 1D to 2D.
Why it matters: The backpropagation analogy gives an intuitive meaning to why transpose operations redistribute values through shared weights.

Key terms: transposed convolution, fractionally strided convolution, transposed weight matrix, upsampling, forward convolution, downsampling, valid padding, non-invertible, information loss, backpropagation

10.5.1 Application of transposed convolution: Autoencoders and embeddings

This section explains how an autoencoder uses a convolutional encoder and a transposed-convolution decoder to learn compact embeddings by reconstructing its inputs.

Embeddings and convolutional encoders: Describe an embedding as a smaller descriptor vector that retains essential features of a class while discarding variable aspects, and explain that an encoder neural network maps an input image to this embedding using convolution layers.
Why it matters: Embeddings let a network represent the important content of an input in a compact form rather than preserving every detail.
Autoencoder reconstruction training: Explain that an autoencoder pairs an encoder with a decoder: the encoder produces a smaller embedding, and the decoder applies transposed convolutions to reconstruct a tensor the same size as the input. Describe training as minimizing a reconstruction loss between the original and reconstructed input while learning both networks' weights end to end.
Why it matters: This explains why transposed convolution is useful: it expands a compact representation back into an input-sized reconstruction.
Bottleneck feature learning and compression: Explain why an embedding with fewer degrees of freedom than the input prevents perfect reconstruction and forces an autoencoder trained on examples of one class to retain features common to that class. Describe how this can discard background or other variable details and make the embedding a compact compression of the objects of interest.
Why it matters: The limited-size embedding is what encourages the model to preserve recurring, useful structure instead of merely retaining the whole input.

Key terms: embedding, descriptor vector, encoder, convolution layers, autoencoder, decoder, transposed convolution, reconstructed input, reconstruction loss, end-to-end learning

10.5.2 Transposed convolution output size

This subsection explains that a transposed convolution’s output size is found by inverting the ordinary convolution output-size equation and illustrates it with a numerical example.

Transposed convolution output size: Describe how to obtain a transposed convolution output size by inverting equation 10.8, and state the example in which stride 1, valid padding 0, input size 3, and kernel size 3 produce output size 5.
Why it matters: Knowing the output dimensions lets you design transposed-convolution layers that expand data to the intended size.

Key terms: transposed convolution, output size, stride, valid padding, kernel

10.5.3 Upsampling via transpose convolution

This section explains how transpose convolution upsamples low-resolution feature maps, especially in decoders, using learned mappings rather than fixed interpolation.

Learned transpose-convolution upsampling: Explain why transpose convolution can be used to upscale a low-resolution input and how it differs from nearest-neighbor, bilinear, or bicubic interpolation. State that interpolation uses a predefined mapping, whereas transpose convolution learns how to distribute input values over a higher-resolution output during training.
Why it matters: Learned upsampling lets a network optimize image reconstruction instead of relying on a fixed resizing rule.
Decoder reconstruction role: Describe the encoder-decoder motivation for upsampling: an encoder can reduce a high-resolution image to a lower-resolution embedding through convolution and pooling, while a decoder upsamples that embedding to reconstruct the original image.
Why it matters: This explains why increasing spatial resolution is a necessary operation in reconstruction networks.
Transpose-convolution tensor layouts: State the input and kernel tensor shapes for 2D transpose convolution: the input is N × Cin × H × W and the kernel is Cin × Cout × kH × kW. Distinguish this kernel layout from regular 2D convolution, whose kernel is Cout × Cin × kH × kW.
Why it matters: Correct channel ordering is essential for specifying and implementing transpose-convolution layers.
Stride-two upsampling example: Describe the section's example in which a 1 × 1 × 2 × 2 input and a 1 × 1 × 2 × 2 kernel, using transpose convolution with stride 2, produce a 1 × 1 × 4 × 4 output.
Why it matters: The example makes the spatial expansion produced by stride-two transpose convolution concrete.

Key terms: transpose convolution, upsampling, nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, learned mapping, autoencoder, encoder, decoder, embedding

10.6 Adding convolution layers to a neural network

This section explains that practical convolutional networks learn convolution weights through backpropagation rather than using manually chosen filters, and introduces the setup required to add such layers.

Learned convolution weights: Explain why convolution weights in a real neural network are learned rather than manually set, contrasting them with the custom weights used earlier for conceptual demonstrations.
Why it matters: Learned filters let a convolutional network adapt its feature detection to the training task.
Convolution layer setup: Describe what is specified when setting up a convolutional neural network layer: its dimensions and initialized weight values, not its final learned weights.
Why it matters: This distinction tells you what a programmer configures versus what training determines automatically.
Backpropagation parameter updates: Trace the role of backpropagation in updating initialized convolution weights during training, including that PyTorch performs the updates behind the scenes after the loss.backward() step while allowing weights to be inspected.
Why it matters: Understanding where parameter updates occur helps you distinguish model definition from model training in PyTorch.

Key terms: custom weights, convolution weights, loss minimization, backpropagation, dimensions, weight initialization, convolution layer, neural network, loss.backward(), weight values

10.6.1 PyTorch- Adding convolution layers to a neural network

This subsection shows the PyTorch structural pattern for placing multiple 2D convolutional layers inside a neural-network class.

PyTorch CNN class structure: Describe how a convolutional neural network is structured in PyTorch: it subclasses torch.nn.Module, creates its layers in __init__(), and defines computation in forward().
Why it matters: This is the basic code organization needed to turn convolutional layers into a complete neural network.
Sequential convolutional layer chaining: Explain that torch.nn.Sequential chains layers so that each layer's output becomes the next layer's input, and state that each torch.nn.Conv2d() object represents one convolutional layer in the example network.
Why it matters: Layer chaining is how individual convolution operations become stages of a larger network.

Key terms: torch.nn.Module, __init__(), forward(), torch.nn.Sequential, torch.nn.Conv2d(), convolutional layer

10.7 Pooling

This section explains how pooling downsamples convolutional feature maps to make CNNs more robust to small feature-location changes.

Pooling and translation invariance: Explain why pooling is used after convolution layers: convolutions are sensitive to feature location, while downsampling makes important features at slightly different positions align more closely in a lower-resolution map. Relate this robustness to local translation invariance.
Why it matters: Pooling helps a CNN recognize a feature even when its position changes slightly in a real image.
Local pooling operations: Describe how a pooling layer slides a small filter across a feature map and replaces each local patch with a summary value. Distinguish max pooling, which selects the patch maximum, from average pooling, which computes the patch average.
Why it matters: Knowing what each pooling operation preserves helps you interpret how a CNN transforms its feature maps.
Pooling output resolution: State how pooling kernel size and stride determine output feature-map resolution. Describe that a 2 × 2 kernel with stride 2 halves each feature-map dimension, whereas a 3 × 3 kernel with stride 3 reduces each dimension to one-third.
Why it matters: Kernel size and stride let you control how much spatial detail pooling removes.

Key terms: pooling layers, downsampling, feature map, local translation invariance, pooling layer, pooling operation, local patch, max pooling, average pooling, kernel size

10.8 Summary

This summary recaps how convolutions, transposed convolutions, and pooling process local patterns and change the spatial size of 1D, 2D, and 3D data.

Where this is used later

CNNs for image classification: LeNet (chapter 11) · LeNet uses convolutional layers to extract image features for classification.
Toward deeper neural networks (chapter 11) · Deeper image networks extend hierarchical feature extraction through additional layers.
Autoencoders (chapter 14) · Autoencoders use learned network layers, including the chapter's convolutional tools, to represent and reconstruct data.
PyTorch- Implementing LeNet for image classification on MNIST (chapter 11) · It prepares the reader to implement convolutional layers in LeNet.
Convolutional backbone (chapter 11) · A convolutional backbone depends on correctly structured convolution inputs and channels.
Geometric view of latent spaces (chapter 14) · Latent spaces are introduced as the geometric setting for compact representations such as embeddings.
(chapter intro) (chapter 14) · The use of transposed convolution in autoencoders and embeddings.
VAE training, losses, and inferencing (chapter 14) · VAE training also uses reconstruction-oriented losses alongside additional terms.

Concepts explained in this chapter

convolutional layerA neural-network layer that applies convolution operations to an input, commonly used to extract local features from images.
convolution kernelA learned kernel applied in a convolution layer to detect local spatial patterns in images.
strideThe step size with which a convolutional kernel moves across the input, affecting the output size in convolution and transposed convolution.
arrayAn indexed sequence of values, here used to represent a signal processed by a convolution.
paddingThe treatment of positions where a convolution kernel extends beyond the input array, such as stopping early or pretending missing values are zero.
transposed convolutionA convolution-like operation often described via the transpose of the convolution matrix, used here as a motivation for understanding the matrix form of convolution.
kernel sizeThe spatial extent of the sliding window used by a pooling layer, determining the patch size summarized at each step.
zero paddingA convolution setting that extends the input with zeros so the kernel can produce outputs near the boundary.
embeddingAn ordered mathematical object representing a point or direction in a coordinate space.
autoencoderA neural network model trained to compress inputs into a latent representation and reconstruct the original inputs from that representation.
spatio-temporal volumeA three-dimensional tensor representing video data across space and time.
convolutional neural networkA neural network that uses convolutional layers to process structured grid data such as images.
local averagingA smoothing technique that reduces short-scale fluctuations by averaging nearby values in one dimension.
edgeA boundary-like location in an input array where values change abruptly, producing a strong convolution response here.
valid paddingA convolution padding scheme that adds no extra border values around the input, so the kernel only applies where it fully fits inside the input.
channelA channel dimension in a tensor that separates different input feature maps or image channels.
poolingA downsampling operation on feature maps that reduces spatial resolution, often by local averaging or max selection.
max poolingMax pooling is an aggregation operation that replaces each local or grouped region with its maximum value, often to downsample a representation.
matrix-matrix multiplicationThe result of multiplying two compatible matrices, here discussed under the rule that transposition reverses the order of factors.
parameterThe tunable values of a model estimated from data.
PyTorchAn open-source machine learning library used here to provide the chapter's executable code examples and automatic gradient computation.
decodeThe reverse mapping that reconstructs data from a reduced representation.
encoderThe neural-network component that maps an input to a latent-space distribution.
objective functionA function assigning a scalar penalty to model parameters or predictions.
reconstructionThe process of producing an output intended to match the original input from a compressed representation.
local patternA pattern that is confined to a small neighborhood of the input and can be captured by a local operator such as convolution.
feature mapA spatial array of activation values produced by applying a convolution or pooling operation over an input image or prior layer output.
pixelA position in a digital image whose stored value corresponds to local brightness or color, used here across time in video frames.
backpropagationThe PyTorch method call that triggers backpropagation to compute gradients from the loss.
forwardThe PyTorch module method that defines how inputs are transformed into outputs during a forward pass.
torch.nn.ModuleThe PyTorch base class that neural network models subclass to define learnable modules and a forward pass.
torch.nn.SequentialThe PyTorch container that chains modules in order so the output of one becomes the input of the next.
linear layerA linear layer is a neural-network layer that maps input features to outputs by a learned affine transformation, typically connecting each output to every input feature.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.