Chapter 10: Interpreting what ConvNets learn
The chapter introduces why interpreting ConvNet decisions matters and why ConvNets are particularly amenable to visualization.
After this chapter you will be able to
- Visual nature of ConvNet representations. Explain why the chapter challenges the idea that ConvNets are simply black boxes.
- Feature maps and channels. Explain what intermediate activations are and why each channel of a feature map is displayed as a separate 2D image.
- Increasing abstraction with depth. Explain how the displayed activations change across layers and what that reveals about information retained by the network.
- Filter visualization by gradient ascent. Explain how optimizing an input image can reveal the visual pattern that activates a chosen ConvNet filter.
- Differentiable model calls. Distinguish model(x) from model.predict(x) when obtaining outputs for filter visualization.
- TensorFlow gradient computation. Explain that in TensorFlow, the loss is computed inside a GradientTape scope to retrieve its gradients, and that a @tf.function decorator is used to speed up computation.
- PyTorch input-image gradients. Explain how this example obtains the gradient of the loss with respect to the input image in PyTorch and why it does not reset the gradients.
- Filter pattern visualization. Describe how a filter index is used to generate a pattern that maximizes that filter’s activation in a target layer, and why the resulting tensor needs post-processing before display.
- Increasing filter complexity. Explain what collections of ConvNet filters reveal about layer representations, from early edges and colors through intermediate textures to higher-level natural-image textures.
- Class activation heatmaps. Explain what a class activation heatmap shows and why it is useful for interpreting a ConvNet classification.
- Grad-CAM channel weighting. Describe how Grad-CAM combines convolutional feature maps and class gradients, at the intuitive level given here.
- Top class activation loss. Explain how the JAX version defines a separate loss function that takes the final layer’s output, selects the activation channel for the top predicted class, and uses that activation as the loss for computing a gradient.
- Class activation heatmap display. Describe how the section obtains and displays a class activation heatmap: pool and importance-weight the gradient tensor, normalize the heatmap to the range 0–1, and superimpose it on the original image.
- Elephant prediction interpretation. Explain what the visualization can reveal about the African elephant prediction and the elephant’s location, including the suggested significance of the calf’s strongly activated ears.
- Semantic structure of latent spaces. Explain that a model embeds inputs as points in a latent space, where nearby points represent semantically similar inputs and the separation of classes enables classification.
- Layer activations as coordinates. Describe how to inspect a layer’s latent space by recording its activations for a set of inputs and plotting the resulting points. Explain why a layer just before classification is especially interesting.
- Limits of two dimensional projection. Explain why high-dimensional activation coordinates need projection for a 2D visualization, why a manifold makes projection feasible, and why the resulting plot cannot preserve all information.
What it builds on
- Introduction to ConvNets (chapter 8) · Assumes familiarity with ConvNets as image-processing models.
- The convolution operation (chapter 8) · The role of convolution layers in a ConvNet.
- The Functional API (chapter 7) · Building Keras models with specified inputs and outputs.
- Building your model (chapter 8) · The ConvNet whose saved model is used here.
- Stochastic gradient descent (chapter 2) · Gradient-based optimization provides the basis for changing the input to increase an objective.
- Feature extraction with a pretrained model (chapter 8) · Feature extraction uses outputs from a pretrained model.
- Inference: Using a model after training (chapter 3) · Inference retrieves outputs from a trained model.
- Derivative of a tensor operation: The gradient (chapter 2) · The gradient is the tensor being normalized.
Section by section
Introduction
The chapter introduces why interpreting ConvNet decisions matters and why ConvNets are particularly amenable to visualization.
- Classifier decision interpretability: Explain why a computer vision classifier’s decision may need interpretation, especially when it complements human expertise.
Why it matters: Understanding why a model made a decision helps people assess its output when their own expertise is involved. - Visual nature of ConvNet representations: Explain why the chapter challenges the idea that ConvNets are simply black boxes.
Why it matters: This is why visualizations can offer insight into what a ConvNet has learned.
Key terms: interpretability, classifier, black boxes, representations, visual concepts, visualization
10.1 Visualizing intermediate activations
The section shows how to display a ConvNet’s intermediate activations and use them to see representations become more abstract with depth.
- Feature maps and channels: Explain what intermediate activations are and why each channel of a feature map is displayed as a separate 2D image.
Why it matters: Looking at channels separately makes the features detected at each layer visible. - Multi-output activation model: Describe how the section obtains the activations to visualize from the saved ConvNet.
Why it matters: One multi-output model provides the layer outputs needed for the visualization. - Increasing abstraction with depth: Explain how the displayed activations change across layers and what that reveals about information retained by the network.
Why it matters: The visualizations illustrate how a deep network distills an image into information useful for classification.
Key terms: intermediate activations, feature maps, channels, Keras model, multi-output model, layer activations, edge detectors, sparsity, abstract, information distillation pipeline
10.2 Visualizing ConvNet filters
This section explains how to visualize what a ConvNet filter responds to by optimizing an input image, and identifies the model calls and gradient handling needed for that process.
- Filter visualization by gradient ascent: Explain how optimizing an input image can reveal the visual pattern that activates a chosen ConvNet filter.
Why it matters: It connects an abstract learned filter to an image you can inspect. - Specific layer feature extractor: Describe how the section obtains a chosen Xception layer’s output and prepares inputs for that model.
Why it matters: The feature extractor supplies the particular layer output whose filter activation will be maximized. - Differentiable model calls: Distinguish model(x) from model.predict(x) when obtaining outputs for filter visualization.
Why it matters: Using predict() here would prevent the gradient calculation needed to change the image. - Gradient magnitude normalization: Explain why the image-update gradient is divided by its L2 norm.
Why it matters: It explains why update size is controlled while the image is optimized.
Key terms: filters, gradient ascent in input space, activation, Xception, Functional API, feature extractor, model-specific image preprocessing, model.predict(x), model(x), differentiable
10.2.1 Gradient ascent in TensorFlow
The subsection identifies how TensorFlow obtains the gradients needed for filter visualization and notes a way to speed up the computation.
- TensorFlow gradient computation: Explain that in TensorFlow, the loss is computed inside a GradientTape scope to retrieve its gradients, and that a @tf.function decorator is used to speed up computation.
Why it matters: This is the TensorFlow setup for obtaining the gradients used in filter visualization.
Key terms: GradientTape, loss, gradients, @tf.function
10.2.2 Gradient ascent in PyTorch
This subsection shows how to obtain input-image gradients in PyTorch and why they do not need to be reset in this example.
- PyTorch input-image gradients: Explain how this example obtains the gradient of the loss with respect to the input image in PyTorch and why it does not reset the gradients.
Why it matters: This explains both where the input-image gradient comes from and why the example can omit a reset step.
Key terms: loss.backward(), image.grad, input image, gradients
10.2.3 Gradient ascent in JAX
10.2.4 The filter visualization loop
This section turns filter-activation patterns into displayable images and uses them to show how ConvNet filters become more complex in higher layers.
- Filter pattern visualization: Describe how a filter index is used to generate a pattern that maximizes that filter’s activation in a target layer, and why the resulting tensor needs post-processing before display.
Why it matters: It explains what the generated pattern represents and why it cannot simply be displayed as-is. - Increasing filter complexity: Explain what collections of ConvNet filters reveal about layer representations, from early edges and colors through intermediate textures to higher-level natural-image textures.
Why it matters: Comparing visualizations across layers makes the progression of learned visual features visible.
Key terms: filter index, target layer, activation, post-process, filter banks, directional edges, colors, textures
10.3 Visualizing heatmaps of class activation
The section introduces class activation heatmaps, explains the intuition behind Grad-CAM, and sets up an Xception example for locating image regions relevant to a prediction.
- Class activation heatmaps: Explain what a class activation heatmap shows and why it is useful for interpreting a ConvNet classification.
Why it matters: A heatmap connects a class prediction to locations in the image rather than leaving the prediction as a single label. - Grad-CAM channel weighting: Describe how Grad-CAM combines convolutional feature maps and class gradients, at the intuitive level given here.
Why it matters: This explains why Grad-CAM’s highlighted regions are tied to a chosen class, not just to strong feature activations. - Xception Grad-CAM setup: Trace the model components and input preparation described for applying Grad-CAM to the elephant image.
Why it matters: The setup makes both the last convolutional activations and the class prediction available for the Grad-CAM gradient.
Key terms: class activation map (CAM), class activation heatmap, output class, Grad-CAM, output feature map, channel, gradient, Xception, classification head, KerasHub ImageClassifier
10.3.1 Getting the gradient of the top class: TensorFlow version
10.3.2 Getting the gradient of the top class: PyTorch version
10.3.3 Getting the gradient of the top class: JAX version
This subsection explains how the JAX version uses the top predicted class’s activation as a loss for gradient computation.
- Top class activation loss: Explain how the JAX version defines a separate loss function that takes the final layer’s output, selects the activation channel for the top predicted class, and uses that activation as the loss for computing a gradient.
Why it matters: Choosing the top-class activation as the loss makes it possible to calculate a gradient for that prediction.
Key terms: JAX, loss computation function, final layer’s output, top predicted class, activation channel, gradient
10.3.4 Displaying the class activation heatmap
This subsection turns class-activation information into a viewable heatmap and uses it to interpret an African elephant prediction.
- Class activation heatmap display: Describe how the section obtains and displays a class activation heatmap: pool and importance-weight the gradient tensor, normalize the heatmap to the range 0–1, and superimpose it on the original image.
Why it matters: The overlay connects the heatmap’s activations to recognizable parts of the picture. - Elephant prediction interpretation: Explain what the visualization can reveal about the African elephant prediction and the elephant’s location, including the suggested significance of the calf’s strongly activated ears.
Why it matters: The highlighted regions offer a visual clue to what influenced the classification, without proving the network’s exact reasoning.
Key terms: gradient tensor, pooling, importance weighting, heatmap of class activation, normalize, superimposes, class activation heatmap, strongly activated, African elephant, Indian elephants
10.4 Visualizing the latent space of a ConvNet
The section explains how a ConvNet’s layer activations can be projected into two dimensions to inspect its latent space, and why that view is necessarily imperfect.
- Semantic structure of latent spaces: Explain that a model embeds inputs as points in a latent space, where nearby points represent semantically similar inputs and the separation of classes enables classification.
Why it matters: This gives meaning to the positions and groupings you see in a latent-space plot. - Layer activations as coordinates: Describe how to inspect a layer’s latent space by recording its activations for a set of inputs and plotting the resulting points. Explain why a layer just before classification is especially interesting.
Why it matters: Choosing a layer determines which stage of the model’s representation you are inspecting. - Limits of two dimensional projection: Explain why high-dimensional activation coordinates need projection for a 2D visualization, why a manifold makes projection feasible, and why the resulting plot cannot preserve all information.
Why it matters: A 2D plot can reveal structure, but it should not be mistaken for a perfect picture of the original space.
Key terms: latent space, manifold, semantically similar inputs, activations, latent manifold, classification layer, projection, intrinsic dimensionality
10.5 Summary
Where this is used later
- Deep learning models are highly sensitive to phrasing and other distractors (chapter 19) · Gradient ascent in input space for ConvNet filter visualization.
Concepts explained in this chapter
- activationThe numeric value of a model activation.
- convolution kernelThe learned weight tensor used by a convolution to transform local input patches.
- class activation heatmapA visualization technique that highlights image regions relevant to a specific predicted class by producing class-specific activation heatmaps.
- interpretabilityThe degree to which a model's behavior can be understood and interpreted by humans.
- feature mapA spatial grid of activations produced by a convolutional or pooling layer, where each channel encodes the presence of a learned feature over locations.
- digital imageData in which each example is a 2D image with one or more color channels, typically stored as a rank-3 tensor per sample.
- convolutional neural networkA neural network built from convolution and pooling layers that processes data with local spatial structure.
- JAXA numerical computing and machine learning library for high-performance array computation, automatic differentiation, and compilation.
- .gradA PyTorch tensor attribute where accumulated gradients are stored for tensors that track gradients.
- backpropagationA PyTorch tensor method that triggers backpropagation from a loss tensor and fills gradients on participating tensors.
- channelA component of a layer output corresponding to one class or feature channel, used here to select the top predicted class.
- tf.GradientTapeTensorFlow’s context object for recording tensor operations so gradients can later be computed from the recorded graph.
- tf.functionA TensorFlow decorator that compiles a Python function into a faster executable form, typically graph mode.
- jax.gradA JAX function that creates a gradient-computing function for differentiable inputs and outputs.
- PyTorchA Python machine-learning library used here as one of the frameworks whose code can run element-wise operations on GPUs; only PyTorch is actually a library entry in the glossary, but the section groups it with JAX and TensorFlow as frameworks.
- gradientThe generalization of a derivative to a tensor-valued function, representing the local rate and direction of change with respect to each input component.
- manifoldA lower-dimensional geometric subspace that is locally similar to Euclidean space, used here to describe the structure of valid data samples.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.